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@ N A T l O N A L U Ji B A N L t AG L? E 



riNIFL 

National Institute for Literacy 



Equipped for the Future — every child, every adult, every community. Nothing better 
defines the partnership between the National Urban League and the National Institute for 
Literacy. Our shared commitment is to assure that all young people, adults and communi- 
ties have the skills, the knowledge and the resources they need to be equipped — not just 
for today — but for the future. For this very reason, we are working together to make a real 
difference in the quality of life for all Americans. 

Ten years ago, Urban League affiliates — in Boston, Hartford, Chicago and Los Angeles — 
helped NIFL define what adults needed to know and be able to do to be better equipped for 
the future. NIFL used that information to build adult learning standards and an approach to 
instruction designed to help adults achieve their goals. This Equipped for the Future (EFF) 
Handbook for Program Improvement is a product of collaboration between NIFL and the 
Urban League to create standards-based tools that will help our programs achieve results 
that matter. 



This EFF Handbook for Program Improvement guides program providers through a process 
of clarifying goals, refining program components, and evaluating program data. This 
process assists programs to more effectively tell the story about how adult education is help- 
ing community members move towards getting and keeping better jobs for a better future. It 
is our hope that this initial publication helps to address the educational needs of the diverse 
communities that the Urban League and the National Institute for Literacy are committed 
to serving. 




Marc Morial 

President and Chief Executive Officer 
National Urban League 



Sandra Baxter 

Interim Director 

National Institute for Literacy 
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The target audience for this handbook includes administrators and other members of 
program improvement teams in community-based organizations that want to use Equipped 
for the Future as part of their efforts to improve program quality. While it will be of 
interest to others, the handbook targets administrators because of their specific roles as 
advocates for program change and because of their responsibility to ensure the delivery 
of quality organizational services. 
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Background 



E quipped for the Future is a standards-based approach to 
reform in the adult literacy and basic education system. It has 
been developed to strengthen the ability of adult education 
providers to improve their programs in order to better meet 
the needs of adult learners and the wider community. Standards-based 
reform is based on content standards that represent a consensus on what 
is important for students to know and be able to do. These standards set 
the definition of quality, and every component of a 
program should contribute to enabling students to 
achieve these standards. In standards-based reform 
we use the term aligned to describe how instruc- 
tion, assessment, and accountability all focus on 
the same things so what we teach is what we assess 
and what we are held accountable for. This effort to 
use standards to align all parts of a program is the 
hallmark of standards-based reform and is critical 
to quality adult programs across the country. 



Standards-based reform is 
based on content standards 
that represent a consensus of 
what is important for students 
to know and be able to do. 
These standards set the 
definition of quality, and every 



What makes Equipped for the Future an especially 
useful vehicle for standards-based reform is that 
the EFF Standards reflect two critical perspectives 
on what results matter: the perspective of policy 
makers who support our programs and the per- 
spective of adult learners who attend our programs. 



component of a program 
should contribute to enabling 
students to achieve these 
standards. 



EFF Standards were developed to help us define — as a nation — what 
adults need to know and be able to do in order to meet the National Adult 
Literacy and Lifelong Learning Goal. This policy goal, developed jointly by 
the President and governors and affirmed by the Congress of the United 
States, defines what state and national policy makers think are important 
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for adult learning programs to achieve: it focuses 
on adults’ ability to exercise the rights and 
responsibilities of citizenship, help their children 
succeed in school, and be productive workers 
who can compete in the global economy. 

In defining what adults need to know and be able 
to do to achieve this policy goal, the National 
Institute for Literacy — which coordinated the 
development of EFF — went first to adult learners 
across the country and asked them to help us understand what these three 
roles — citizen, family member, worker — looked like in their daily lives 
and what they needed to know and be able to do in order to carry out 
these roles. The responses of adults gave us the second critical part of our 
framework — adult purposes for learning. Since the framework we used to 
develop EFF Standards includes a focus on both adult learner purposes 
and policy maker goals, it enables us to pay attention to both these critical 
customers of adult learning programs. Aligning programs toward achiev- 
ing EFF Standards helps sharpen our focus on learner goals and at the 
same time achieve results that are important to policymakers. This is why 
we call EFF a customer- driven framework. If we meet EFF Standards, we 
are being accountable both to our learners and to our funders. 



National Adult Literacy and 
Lifelong Learning Goal (Goal 6) 

Every adult American will 
be literate and possess the 
knowledge and skills necessary 
to compete in a global economy 
and exercise the rights and 
responsibilities of citizenship. 
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T his handbook is designed to help community-based organi- 
zations that provide adult education improve their program 
quality in order to better achieve results that matter. The tar- 
get audience for this handbook includes administrators and 
other members of program improvement teams in community-based 
organizations that want to use Equipped for the Future as part of their 
strategy for program improvement. While it will be of interest to others, 
the handbook targets administrators because of their specific roles as 
advocates for program change and because of 
their responsibility to ensure the delivery of quali- 
ty services. 

The handbook uses Equipped for the Future to 
frame a program improvement process. Carrying 
out the activities in the handbook will 

• enhance an organization’s understanding of 
EFF and how to use the EFF Standards and 
tools to improve quality 
• help the organization identify strengths as 
well as areas of practice to change 
• support the development of a plan for implementing changes 
• support the documentation of the change process and results. 

The National Urban Teague and its affiliates in Columbus, Ohio, and 
Charlotte, North Carolina, worked with Equipped for the Future to devel- 
op this handbook in order to enhance the quality, consistency, and 
accountability of their programs. The processes described in the hand- 
book will increase organizations’ abilities to meet funders’ accountability 
guidelines and better meet the needs of their students and communities. 



The processes described in 
the handbook will increase 
organizations’ abilities to 
meet funders’ accountability 
guidelines and better meet the 
needs of their students and 
communities. 
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Program Improvement Focused on Quality and Results 

Although there are many approaches to implementing a program 
improvement process in education, all include a systematic approach to 
defining program goals and to evaluating how well each program compo- 
nent contributes to achieving these goals. 

This Equipped for the Future approach to quality is structured by five EFF 
Program Practices based on the theoretical foundations of EFF: 

• A purposeful and transparent approach to education 

• A contextualized approach to curriculum and instruction 

• An approach to teaching and learning based on building expertise 

• An approach to assessment based on cognitive science 

• A systematic approach to program improvement. 

These Program Practices provide guideposts by which programs can 
assess progress and make plans for improvement based on EFF. 

Using EFF as a framework and guide can lead to improved outcomes for 
students, teachers, and programs: 

• Students will be better able to articulate their goals and what they 
need to learn to reach those goals. 

• Teachers’ planning and instruction will be more focused on student 
goals and structured by the adult learning theory that is the basis of 
EFF. 

• Programs will be better able to align program components, describe 
their student progress in measurable, real-world results that can be 
understood by their constituencies, and will be able to use data about 
student progress in a process of ongoing program improvement. 

Program improvement processes involve organizations in an ongoing 
process of assessing current practice, planning needed changes, imple- 
menting planned changes, collecting data on the outcomes of these 
changes, evaluating the results, and based on these results, planning the 
next cycle of improvement efforts. This handbook guides organizations 
through four phases of using EFF tools and processes to carry out a pro- 
gram improvement process: 

Phase 1 is a consideration of preliminary issues and organizational 
requirements for engaging in an improvement process. In Phase 1, the 
people leading the EFF quality process prepare for their role by examining 
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the reasons for launching the process, their role in facilitation, considering 
issues of team formation, and planning for Phase 2. They learn about and 
use some of the EFF Standards in this process. 

Phase 2 is an examination of the adult education program using the 
EFF program practices to look at “what is” and “what could be” to iden- 
tify needed changes. In Phase 2, the program improvement team takes 
part in a series of five sessions, using EFF concepts to examine the adult 
education work of their organization. These activities focus on the five 
EFF Program Practices. 

Phase 3 is a prioritization and planning process to determine which 
changes to implement and what modifications in current program 
practices should be made to support those changes. In Phase 3, the pro- 
gram improvement team will use what they have learned in Phase 2 to 
examine possible program changes, to choose the most important, and to 
plan implementation of these changes. 

Phase 4 includes the implementation, documentation, and evaluation 
of changes and modifications. In Phase 4 the program implements the 
changes planned in Phase 3, and the program 
improvement team documents and evaluates these 
changes. 

These phases are part of an ongoing iterative process 
of program improvement using Equipped for the 
Future. 

More About Equipped for the Future 
EFF was initiated in 1993 in response to National Edu- 
cational Goal 6: 

Every adult American will be literate and pos- 
sess the knowledge and skills necessary to com- 
pete in a global economy and exercise the rights 
and responsibility of citizenship. 



THIS HANDBOOK INCLUDES 

• Steps in Phase 1 to prepare 
for the process. 

• A facilitator’s guide for 
team sessions in Phase 2 
to examine “what is.” 

• Tools for developing 
specific plans for program 
improvement in Phase 3. 

• Tools to be used to docu- 
ment the implementation 
and results of program 
improvements in Phase 4. 

• Resource List of other 
materials that support these 
processes. 
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By clearly defining what adults need to know to achieve this goal, EFF 
makes it possible for the adult literacy system to focus on measurable 
standards so that we can determine progress toward achieving Goal 6. 

In research with adult students, EFF has identified four purposes for 
learning: 

• to gain access to information and resources to orient themselves in 
the world 

• to give voice to ideas and opinions with the confidence that they will 
be heard 

• to take independent action to solve problems and make decisions 

• to keep learning to build a bridge to the future in a rapidly changing 
world. 

EFF helps organizations enable adults to achieve these four purposes in 
their roles as citizens, workers, and family members by providing 

• a clear set of 16 Content Standards that describe what adults need to 
know and be able to do, 

• a framework and tools that teachers and programs can use to link 
curriculum, instruction and assessment, 

• an assessment framework that can be used to support program 
improvement and accountability. 

The book, Equipped for the Future Content Standards: What Adults Need to 
Know and Be Able to Do in the 21st Century (Stein, 2000) summarizes EFF 
history and goals, defines the key elements of EFF, defines the EFF Con- 
tent Standards, and gives examples of how the EFF Content Framework 
has been used. 

The five EFF Program Practices reflect the theoretical foundations of EFF. 
These practices are explained in Results That Matter: An Approach to Pro- 
gram Quality Using Equipped for the Future (Bingman and Stein, 2001). 
See the Resource Fist in the Appendix for ordering information. 

To begin exploring Equipped for the Future and for information on EFF 
materials and other EFF resources, visit the Web site: 

www.nifl.gov/linc/collections/eff/eff.html 
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More About Program Improvement 

There are many approaches used to structure program and organizational 
improvement. In the processes described in this handbook, you will learn 
about EFF program practices that will help you carry out a systematic 
analysis of your program and identify changes you might want to imple- 
ment. While the EFF approach to quality presented in this handbook can 
be used alone, it can also be combined with other quality improvement 
systems that focus on the organization as a whole. One of the most widely 
used is the Baldrige National Quality Program in Education developed in 
the U.S. Department of Commerce in 1988. The Baldrige National Quali- 
ty Program is based on seven criteria for performance: 

• Feadership 

• Strategic Planning 

• Student, Stakeholder, and Market Focus 

• Information and Analysis 

• Faculty and Staff Focus 

• Process Management 

• Organizational Performance Results 

These criteria can be used to assess performance on a wide range of key 
indicators and can help align resources; improve communication, produc- 
tivity, and effectiveness; and achieve strategic goals. 

Please refer to the Baldrige National Quality Program: Education Criteria 
for Performance Excellence for additional information. See the Resource 
Fist for contact information. 
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The Four Phases of Program Improvement 



PHASE 1: Getting Started 

• Deciding to do program improvement 

• Building an administrative team 

• Understanding roles and responsibilities 

♦ 

PHASE 2: Examining “what is” in your program 
using the five EFF Program Practices 



The program improvement team participates in a series 
of five sessions learning about and using EFF to assess 
their adult education program. 

\ 

PHASE 3: Planning changes for program improvement in: 



Staff 



Procedures 



Teaching and Learning 



Resources 




PHASE 4: Implementing, Documenting, and Evaluating Changes 

Using EFF tools to make and evaluate planned changes. 



In continuous program improvement, this process is cyclical and 
after changes are implemented and data collected, the team will 
begin a new cycle of assessment, planning, and implementation. 
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T his section of the handbook is written for the person or per- 
sons who will be leading the EFF program improvement 
process in an organization. Phase 1 includes information and 
activities needed to prepare for a program improvement 
process and to form and to lead a program improvement team. This sec- 
tion explains how to get started and how the EFF Standards support the 
program improvement process. The “Getting Started” phase is based on 
the assumption that the organization is committed to a timely process and 
has identified administrators and/or leaders within the organization who 
understand that leading the process is an essential part of their work. 

In Phase 1, you will consider how your organization gets ready for a pro- 
gram improvement process and consider the questions: 

What do we need to know about EFF and about leading an 
improvement process? 

What do we need to do to ensure a smooth, thoughtful, and 
successful process? 

This handbook provides the administrative team with information, tools, 
activities, and handouts needed to facilitate a process of using Equipped 
for the Future to strengthen your organization. The activities in the hand- 
book can be carried out in a variety of organizational settings, such as staff 
and program meetings, organizational retreats, board meetings, strategic 
planning sessions, etc. Many of the activities in the handbook include 
guiding questions as discussion prompts. Handbook users are encouraged 
to adapt particular activities to meet the needs of diverse organizations 
and programs. 



AN EFF HANDBOOK FOR PROGRAM IMPROVEMENT 

9 



Phase 1: Getting Started 



In Phase 2, “Examining Your Program,” administrators and other team 
members, are introduced through practical activities, to aspects of the five 
program practices and other EFF tools, and examine their program to 
identify changes (documented as Changes to Make) to be implemented 
later. The facilitator and team members will become familiar with EFF 
language and tools through the process of examining current organiza- 
tional and program practices. In Phase 2 of the handbook, you will also 
find several Cases in Point. These are scenarios of everyday situations that 
present themselves in adult education programs that participants use to 
reflect on how EFF might be used. 

Phase 3 is framed using the EFF Content Framework Standards Solve 
Problems and Plan. The program improvement team will prioritize its 
goals and then evaluate the alternatives in the Changes to Make lists gen- 
erated in Phase 2. The team will select the alternative(s) you want to 
implement and develop a plan of objectives and activities to implement 
the chosen alternatives. The change(s) might involve one or all of these 
program aspects: staff, procedures, teaching and learning, and policy. 

Phase 4 is the implementation phase and continues to use components 
from the Solve Problems and Plan Standards. The organization or program 
will implement the plan, document what happens and the results, and use 
this documentation to evaluate what has been done. 



Documenting activity outcomes. Users of this 
handbook are encouraged to document activi- 
ty outcomes and use the results to inform and 
plan for changes in Phase 3 of the program 
improvement process. Fists generated during 
brainstorms, participants’ questions concern- 
ing the program improvement process, sugges- 
tions for changes to improve program quality, 
and reactions to and evaluation of the activities 
are examples of information the facilitator 
should document and maintain. Information captured on flipchart sheets 
during the activity will need to be typed and filed in a folder or three-ring 
binder. Notes taken by the facilitator and participants’ evaluations should 
also be saved. 




This documentation 
symbol will appear 
throughout the guide as a reminder 
to facilitators to take note of and 
document a particular activity’s 
outcomes and to save materials 
created. 
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Steps in Getting Started 

Phase 1 asks the facilitator or facilitation team to take several steps in 
preparation for carrying out the activities in Phases 2-4. In Phase 1 you 
will 

• consider your decision to engage in program improvement 

• become more familiar with the EFF Standards 

• review facilitation skills 

• establish a program improvement team 

• decide on the roles and responsibilities of the team 

• set the schedule 

• determine your communication processes 

• assemble the materials you will need. 

Your own schedule will determine how long Phase 1 will take. 




Deciding to Engage in Program Improvement 

Programs decide to undertake a program improvement process for many 
different reasons, in response to both internal and external factors. Inter- 
nally, it may appear that changes are needed in your program because the 
students are not meeting their goals and are not attending classes. Teach- 
ers may be complaining that the work is too hard and that they see few 
positive results. Or they have seen extraordinary results in one class and 
would like to see other classes improve. Externally, funders are now 
requiring higher levels of accountability and want to see results that justify 
their financial support. Perhaps there is a funding 
source that you have never been able to access 
because your data collection process was not suffi- 
cient; or a funder may be interested in supporting 
your organization through an improvement process. 

Whatever the situation, being intentional about pro- 
gram improvement is a positive first step. 

Fist as many reasons as you can think of for engaging 
in program improvement and save for future use. 



Reasons for engaging in program 
improvement 
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Now that you have identified reasons for pursuing a program improve- 
ment process, consider whether you are thinking of focusing on a single 
program or your entire organization. You want to be clear from the begin- 
ning what you are changing and what the implications might be for the 
entire organization. This handbook will take you through the steps for an 
adult education program pursuing program improvement and interested 
in using the EFF Framework. However, with additional organizational 
development tools this approach can be adapted for improving an entire 
organization. 

The EFF Standards 

As you embark upon this program improvement process, you will need to 
know about team dynamics and development. The EFF Content Stan- 
dards offer important insights and are useful for considering team work 
and other tasks related to a program improvement process. These Stan- 
dards make clear the key elements of the core knowledge and skills adults 
use in their lives. Just as the Standards give students understanding of 
what a skill means, they also help teachers and administrators, and can 
help you, as a facilitator, understand what’s involved in carrying out a par- 
ticular skill. 



REVIEW: The EFF Standards and how they work in the Equipped for the 
Future Content Standards book, pages 17-23. 



The Standards are divided into four categories of skills: 

• Communication Skills 

• Decision-Making Skills 

• Interpersonal Skills 

• Fifelong Teaming Skills. 

Each Content Standard’s page includes the four purposes for learning 
identified by adults, the three roles in which adults function, and the 13 
Common Activities that represent the overlap between the three roles. 
Under each Standard is a list of specific skills and abilities that learners 
need in order to carry out a particular standard. These components sup- 
port learners’ purposes for learning. 
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The following Standards will be especially helpful as you approach the 
tasks in a program improvement process. Each Standard is explained in 
detail in the EFF Content Standards book: 

• The Cooperate with Others Standard (page 41) for understanding 
team dynamics 

• The Reflect and Evaluate Standard (page 51) for knowing how to 
facilitate reflection 

• The Plan Standard (page 39) for learning how to plan and manage 
change. 




Team Facilitation 

Teams share a common purpose; team members work together to realize 
specific goals. As an administrator and leader, it is critical that you orga- 
nize a representative team of staff and stakeholders. Here are some things 
to consider about team dynamics: 

• Stages of team development 

• Types of team behavior 

• Processes for team decision-making 

• Roles of team members. 

All teams or working groups go through stages of team development: 
forming, storming, norming, performing, and adjourning. 

Think about experiences you have had as a team 
member in your organization. “What seemed to 
unfold as the group developed?” Make a bulleted 
list. 



REVIEW: Review the “Stages of Team Development” 
in Appendix A. Compare your “Working with Teams” 
list with the stages of team development. Can you 
identify aspects of any stages on your list? Consider 
how you will take these stages into account as you 
facilitate this process. 



Working with teams 
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Now, look at the Cooperate with Others Standard and think about how it 
is useful to your team building efforts. (See page 41 in the EFF Content 
Standards book.) The Cooperate with Others Standard is located in the 
interpersonal skills category. It has four components of performance: 

• Interact with others in ways that are friendly, courteous, and tact- 
ful and that demonstrate respect for others’ ideas, opinions, and 
contributions. 

• Seek input from others in order to understand their actions and 
reactions. 

• Offer input on one’s own interest and attitudes so that others can 
understand one’s actions and reactions. 

• Try to adjust one’s actions to take into account the needs of others 
and/or task to be accomplished. 

Consider each component and determine how they relate to the list of 
experiences generated earlier. Compare issues of team development with 
components of performance from the standard Cooperate with Others. 

Now, look at the Plan Standard. It is found under the Decision-Making 
Skills category and it has five components of performance: 

• Set and prioritize goals. 

• Develop an organized approach of activities and objectives. 

• Actively carry out the plan. 

• Monitor the plan’s progress while considering any need to adjust 
the plan. 

• Evaluate its effectiveness in achieving the goals. 

The Plan Standard is an important one to consider in this program 
improvement process. The components of performance provide the 
direction that you and other team members will need in order to priori- 
tize new program goals and practices and to implement programs 
changes. Phases 3 and 4 are structured around the Plan Standard. 
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Review the Standard Reflect and Evaluate and its components of perfor- 
mance: 

• Take stock of where one is: assess what one knows already and the 
relevance of that knowledge. 

• Make inferences, predictions, or judgments based on one’s reflec- 
tions. 



Throughout the process the program improvement team will use these 
two components of performance. In Phase 2, the team members will 
repeatedly examine and take stock of current program practices and use 
that data to inform what might improvement the quality of a program. 



The EFF Content Standards offer invaluable 
insights into implementing and managing a pro- 
gram improvement process, so use them often. 



See Appendix A for additional 
resources on group facilitation. 



Establishing the Team 

You will need to establish a program improvement team. Members of the 
team will work together to reflect on current program practices and to 
plan and implement program improvements. Who should be on the 
team? It is important that the team is representative of the organization or 
specific program. Therefore the team might include the following: 

• program director 

• program administrator 

• board member 

• student leader 

• lead teacher. 

The team might also include key stakeholder representatives. For the ini- 
tial phase, you may need eight to twelve people. It will be important to 
have administrators who can influence and advocate for program change. 
For adult education programs, it will be critical to have a lead teacher as 
part of the team. If the teacher works only part-time for the organization, 
it will be important to compensate him or her for time involved in plan- 
ning and other team activities. 
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Roles and Responsibilities 

Establishing clear and manageable roles and responsibilities is critical to 
the program improvement process. The program improvement team will 
be the primary entity responsible for reflecting on current practices and 
planning and implementing new program practices and improvements. 
There is no substitute for the commitment of all team members to the 
process. Since ah teams develop through stages, it might be helpful for 
team members to check in periodicahy throughout the process to reflect 
on the stage in which they are operating. Making explicit what is going on 
keeps the communication channels open so the work can go forward. 

Team members will need to decide what roles and responsibilities are nec- 
essary for at least the initial phases of the process. Consider the following 
questions: 

• Will the team have a team chairperson? 

• If so, will the chairperson be responsible for facilitating the meetings? 

• Who will create the agenda and be responsible for collecting meeting 
evaluations and other information? 

• What are the responsibilities of the team members? 

• What will be the senior or executive director’s role and responsibility 
during this process? 

• Will the team have a recorder or note taker? 

• If so, will the recorder be responsible for taking notes during ah of 
the meetings and activities? 

• Will some roles be rotated among team members? 

Administrator’s Role. There may be several administrators on the team. 
Remember this process focuses on the administrative level and the issues, 
changes, and improvements that align instruction, assessment, and 
accountability. Specifically, the administrator’s role in this process has two 
parts: 1) understanding how to create an organizational environment to 
support teachers in using the EFF Teaching and Learning Cycle and other 
EFF tools to help students meet their goals; 2) learning how to increase 
program performance and accountability. The administrator provides 
leadership for using the EFF Program Practices in a systematic approach 
to program and organizational improvement. 

As team members, administrative staff will be mindful of administrative 
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level tasks and needs throughout the process. It will also be important for 
administrators to understand (and internalize) the EFF Program Practices 
in order to model EFF quality instructional leadership. 

Facilitator’s Role. This handbook is addressed to the facilitator of the pro- 
gram improvement process and particularly to the facilitator of the reflec- 
tion activities in Phase 2. The facilitator’s role may be shared or rotated 
among team members, or one administrator might serve as facilitator 
throughout the process. The facilitator’s role is to 

• understand the overall direction of the process and the goal and 
objectives of the sessions 

• organize materials and handouts for the sessions 

• ensure that everyone participates and no one person dominates the 
meeting 

• focus the energy of the group on the task 

• help resolve conflict 

• understand when decisions are needed and suggest discussion or 
decision-making methods 

• ensure that there is accurate documentation of activity outcomes 
and process evaluation. 

As a team you might want to add to or delete from this list as well as think 
of other roles and responsibilities that are essential to this process. 

Setting the Schedule 

Your organization’s leaders can determine when the time is right for start- 
ing a program improvement process. Remember that the best time for 
implementing change is when the organization is stable and things are 
going smoothly, not when the organization is experiencing uncertainty 
and chaos. If the organization is using the Baldrige National Quality Pro- 
gram framework or some other quality framework, that schedule should 
be taken into account. 

The initial phase of this program improvement process involves a series of 
five sessions, which can span anywhere from a month to three months, 
depending on the organization’s situation and circumstances. The organi- 
zation’s senior directors should have input into the amount of time allo- 
cated for the entire process. The meetings should be held at a convenient 
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time for all team members. There should be enough time set aside to 
comfortably complete all of the steps in a session. 



Communication 

Moving forward with a program improvement process requires effective 
communication throughout all levels of the organization, including 
trustee boards, advisory boards, the entire staff, stakeholders, consumers 
of services, etc. The organization or program should determine how 
extensive this communication needs to be at different points in the 
process. Consider the messages that you want to convey about program 
improvement in your organization and get it out to all concerned parties. 



Materials to Assemble 

As part of your own preparation, you have reviewed EFF publications. 
Every team member should have a copy of the Equipped for the Future 
Content Standards book and Results That Matter. These should be ordered 
a few weeks before you plan to begin Phase 2. Results That Matter can also 
be downloaded from the EFF website. You may want to order copies of the 
“EFF Research to Practice Notes 1-3” for each team member, although you 
can also copy these as handouts from masters in the Appendix. Informa- 
tion for obtaining EFF materials is in the Resource List. 

In addition to EFF materials, you will need to assemble these documents 
from your organization: 

• mission and vision statements 

• brochures, newsletters, recruiting pamphlets 

• samples of tests and forms used to document student progress. 

At every meeting, you will need an easel, flipchart pads, writing paper, 
pens, and large color markers. At the beginning of the guide for the steps 
in each session is a list of specific materials and handouts required for that 
session. The facilitator will need to review and assemble these materials 
before each session. Handout templates used during the activities are in 
Appendix B. 

The chart that follows outlines the materials and handouts needed for 
each session. 
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Sessions Materials Required (handouts in Appendix) Time 



Session One: Introduction to EFF. 

Team members are introduced to the 
program improvement process and 
the EFF Framework, create a meeting 
schedule, clarify their program goals 
and compare them to the four 
purposes which adults identify as 
important reasons for adult 
education, and use EFF Standards 
to better understand the relationship 
between skill building, standards, and 
contextualized teaching and learning. 


• Introduction to EFF handout 3 hours 

• Role Maps or EFF Content Standards publication 

• EFF Content Framework handout 

• Example handout 

• Changes to Make form 

• Copies of organizational mission and vision statement 

• Copies of Goals Grid handout 

• Flipchart sheets with: 1) a reproduction of the Goals Grid, 

2) EFF Four Purposes, and 3) Changes to Make 

• Standards Wheel Poster (See Resource List for ordering information) 

• Session #2 Background and Case in Point 

• Copies of EFF Research to Practice Note 1 


Session Two: A Purposeful 
and Transparent Approach to 
Education. Team members review 
the organization’s stated goals 
through its mission statement and 
published materials, and compare 
those goals with student and staff 
goals. 


• Copies of organizational brochure, recruitment materials, 2-1/2 hours 

newsletters, other publications 

• Changes to Make form 

• Flipchart sheets labeled Changes to Make, Intake Process/ 

Orientation Process 

• Session #3 Background and Case in Point handouts 

• EFF Research to Practice Notes 2 & 3 


Session Three: Building Expertise 
Using a Contextualized Approach 
to Curriculum and Instruction. 
Team members identify contexts of 
students’ lives and consider how the 
EFF Teaching/Learning cycle 
supports students in building 
expertise. 


• Session #3 Background and Case in Point 3 hours 

• EFF Research to Practice Notes 2 &3 

• Flipchart page to list Students’ Real-life Contexts 

• Contextualized Approach to Curriculum and Instruction grid 
on flipchart 

• Constructivist Learning handout 

• EFF Teaching and Learning Cycle handouts 

• EFF Teaching and Learning Grid handouts 

• EFF Teaching and Learning Cycle grid on flipchart 

• Session #4 Background and Case in Point handouts 

• Joan Benz’s articles 


Session Four: Assessment, 
Accountability and Program 

Improvement. Team members 
explore the organization’s 
assessment practices and consider 
how assessment might look if it 
were based on the EFF Assessment 
Framework and review the 
organization’s current documentation 
methods and accountability 
processes. 


• Session #4 Background and Case in Point 3 hours 

• Questions on Assessment handout 

• Current Approach to Assessment handout 

• EFF vs. Traditional Assessment handout 

• Current Approach to Assessment Grid handout 

• Joan Benz’s article on assessment in the EFF classroom 

• Current Approach to Assessment, Documentation Methods 
Grid, Changes to Make on flipcharts 

• Large table for sorting documents 

• Goals list from Session 1 

• Session #5 Background and Case in Point 

• Merrifield article handouts 


Session Five: Program 
Improvement: Accountability, 
Change and Quality. Team builds 
on the organization’s experiences with 
implementing changes and compares 
funders’ accountability requirements 
with current organizational 
accountability processes. 


• Questions about Change handout 2-3/4 hours 

• Flipchart pages from Steps 1 

• Program goals list generated in Session 1 

• Quotes flipchart 

• Merrifield’s article on accountability 

• Changes to Make forms from previous meetings 

• EFF and Baldrige matrix drawn on flipchart sheet 
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T his section of the EFF Handbook for Program Improve- 
ment contains step by step directions for facilitating five 
sessions that take the program improvement team 
through a series of activities in 
which they examine various 
aspects of the program and learn about 
Equipped for the Future. Each session is orga- 
nized in a similar way. 



Within the session 
directions, handouts are 
indicated by this symbol. 



Before each session, all participants should read 
the handouts “Background and Case in Point.” 
The Background for Session One includes an 
introduction to EFF and should be distributed 
to participants a few days before the first session. 
These handouts are reproduced at the beginning 
of each session section as an introduction and 
review for the facilitator. Copies for participants 
are found in the Appendix. 

Each session has goals and objectives. As facili- 
tator, you should review these with team mem- 
bers at the beginning of each session. You may 
want to copy them on a flipchart sheet and post 
during the session. 




Information, ususally 
recorded on flipchart 
pages, is indicated by this 
symbol. 




This documentation 
symbol will appear 
throughout the guide as a reminder 
to facilitators to take note of and 
document a particular activity’s 
outcomes and to save materials 



As you and the team take part in these sessions 
you will generate a variety of information that 

will guide you as you plan and implement changes in your program. This 
information, usually recorded on flipchart pages, should be saved. Be sure 
to label and date each flipchart page. It may be helpful to have this infor- 
mation typed and copied for team members to save in a notebook. 



At the conclusion of Phase 2 your team should be ready to design a pro- 
gram improvement plan in Phase 3. 
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Session One - Introduction to EFF 

Handout 2.1 



HANDOUT 2.1 

Introduction to EFF 

In the initial session of this program improvement process, team members are introduced to EFF as a 
standards-based framework for guiding the delivery of quality adult basic skills education. A quality pro- 
gram is one in which the organizations purposes and goals are clear, so that students and community 
members can understand how a program or organization contributes to the students’ goals and to the 
broader community’s goals. 

EFF was initiated in 1993 in response to National Educational Goal 6: 

Every adult American will be literate and possess the knowledge and skills necessary to 
compete in a global economy and exercise the rights and responsibility of citizenship. 

By clearly defining what adults need to know to achieve this goal, EFF makes it possible for the adult liter- 
acy system to focus on measurable standards so that we can determine progress toward achieving Goal 6. 

In research with adult students, EFF has identified Four Purposes for Learning 

• to gain access to information and resources to orient themselves in the world, 

• to give voice to ideas and opinions with the confidence that they will be heard, 

• to take independent action to solve problems and make decisions, and 

• to keep learning to build a bridge to the future in a rapidly changing world. 

EFF helps organizations enable adults to achieve these four purposes in their roles as citizens, workers, 
and family members by providing 

• a clear set of 16 Content Standards that describe what adults need to know and be able to do 

• a framework and tools that teachers and programs can use to link curriculum, instruction and assess- 
ment 

• an assessment framework that can be used to support program improvement and accountability. 

The book, Equipped for the Future Content Standards: What Adults Need to Know and Be Able to Do in the 
21st Century (Stein, 2000) summarizes EFF history and goals, defines the key elements of EFF, defines the 
EFF Content Standards, and gives examples of how the EFF Framework has been used. 

The five EFF Program Practices reflect the theoretical foundations of EFF. These practices are explained 
in Results That Matter: An Approach to Program Quality Using Equipped for the Future (Bingman and 
Stein, 2001). See the Contacts list in the Appendix for ordering information. 



Session 1 Goals: 

To review primary components of the EFF Framework and the organiza- 
tion’s stated goals through its mission statement and published materials, 
and to compare those goals with student and staff goals. 

Objectives: 

By the end of this session, participants will be able to 

• understand the three role maps and how they help students set goals and 
plan learning 

• understand how the role maps can enhance a purposeful and transpar- 
ent approach to education and enhance program quality 
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• clarify the purposes of the educational pro- 
grams 

• identify specific steps for making the program 
purposeful and transparent, and 

• discuss the EFF four purposes for learning. 

Materials Required : 

• The three Role Maps in the EFF Content Stan- 
dards book. 

• Introduction to EFF handout (2.1) 

• The EFF Content Framework handout (2.2) 

• Handout Example (2.3) 

• Flipchart sheets with EFF Four Purposes, a 
reproduction of the Goals Grid, and Changes 
to Make form 

• Copies of organization’s mission and vision 
statement for each team member and a copy on 
a flipchart sheet 

• Goals Grid handout (2.4) 

• Post-it Notes 

• Standards Wheel Poster (See Resource List for 
ordering information) 

Time Required: 3 hours 



▼ 

Facilitator’s Note: Since this is the 
first session that brings the program 
improvement team together, it will be 
important to review the organization’s 
entire program improvement process 
or the phases of this project and the 
expected outcomes of this project. 
Then explain the goal and objectives 
of this session. Make sure that 
everyone is comfortable with the 
process. Announce any logistical 
issues related to the meeting. As 
the facilitator and team member, 
you will also participate in the 
session activities. Everyone should 
have copies of materials and 
handouts to be used in the session. 
You can create a packet of materials 
for each team member beforehand or 
distribute materials, as necessary, 
during the session. 

In this session, the terms “goal” 
and “purpose” might be used 
interchangeably and might refer to 
students and/or to the program. As 
the facilitator, it will be important that 
you clarify how the terms are being 
used. 



Steps in Session One 

Settle-In and Session Overview - 20 minutes 

Allow time for everyone to settle in and get comfortable, and if necessary 
have team members introduce themselves. 

Explain overall project goals, expected outcomes, and how this project fits 
with your organization’s program improvement process, if any. 

Discuss the schedule for team activities. 
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Invite participants to ask questions about EFF. All questions might not be 
answered at this time; however, encourage participants to make note of 
their questions for later discussions. 



Pass out and use the handout below to briefly review the components of 
the EFF Content Framework. 

Handout 2.2 



HANDOUT 2.2 

The Equipped for the Future Content Framework 



1 Four Purposes 


Role Maps 


Common Activities 


Content Standards 1 


To gain 


Citizens and 


• Gather, analyze and 


Communication Skills 


ACCESS 


Community Members 


use information 


• Read With Understanding 

• Convey Ideas in Writing 






• Manage Resources 


• Speak So Others Can 
Understand 






• Work within the big 
picture 


• Listen Actively 


To give 






Decision-Making Skills 


VOICE 


Parents and Family 


• Work together 


• Solve Problems and Make 
Decisions 




Members 


• Provide leadership 


• Plan 

• Use Math to Solve 






• Guide and support 


Problems and 






others 


Communicate 


To take 




• Seek guidance and 


Interpersonal Skills 


INDEPENDENT 




support from others 


• Cooperate With Others 


ACTION 


Workers 


• Develop and express 


• Guide Others 

• Advocate and Influence 






sense of self 


• Resolve Conflict and 
Negotiate 






• Respect others and 
value diversity 


Lifelong Learning Skills 








• Take Responsibility for 


To build a 




• Exercise rights and 


Learning 


BRIDGE TO 




responsibilities 


• Learn Through Research 


THE FUTURE 




• Create and pursue 


• Reflect and eEvaluate 

• Use Information and 






vision and goals 


Communication 

Technology 






• Use technology 
and other tools to 
accomplish goals 








• Keep pace with 
change 





Step 1: Using Role Maps to Set Goals 
and Plan Learning - 30 minutes. 

Be sure each person has handouts of the role maps or you can refer them 
to role maps in the back of the EFF Content Standards book. In this step 
you will discuss the three EFF Role Maps, decide which one is most rele- 
vant to your personal goals, and explore how the role maps can help stu- 
dents set goals and plan learning. Each team member should review all 
three of the role maps. Point out that 

• The role maps define key activities necessary to fulfill the three primary 
adult roles: 
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1. Effective citizens and community members take informed action to 
make a positive difference in their lives, communities, and world. 

2. Effective parents and family members contribute to building and 
maintaining a strong family system that promotes growth and 
development. 

3. Effective workers adapt to change and actively participate in meet- 
ing the demands of a changing workplace in a changing world. 

• Each role map identifies key activities that are critical to carrying out the 
responsibilities associated with a particular role. Role Indicators found 
in the EFF Content Standards book describe key characteristics of effec- 
tive performance of each activity. 

Pass out and review the example below. Then ask each team members to 
think of a goal that relates to his or her work life. Ask team members to 
read through the Broad Areas of Responsibility and Key Activities on the 
Worker Role Map and figure out where their goals fit. 

Handout 2.3 



HANDOUT 2.3 

EXAMPLE: Goals and Broad Areas of Responsibility 

My personal goal is to create a small, home-based arts business that I can work 
and operate in my spare time. When I looked at the Worker Role Map, I located 
my goal under the Broad Area of Responsibility — “Plan and Direct Personal 
and Professional Growth.” When I looked closer at the Plan and Direct Personal 
and Professional Growth" section of the Role Map, I found the following Key 
Activities: 

• Balance and support work, career, and personal needs. 

• Pursue work activities that provide personal satisfaction and meaning. 

• Plan, renew, and pursue personal and career goals. 

• Learn new skills. 

So, these are four key activities that I need to “perform” so that I’m prepared as a 
small, home-based business “worker.” 




Go around and have each person share his or her goal and how it connects 
to the role maps. Finish this step by discussing and documenting how the 
role maps might help students set goals and plan learning. 
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Step 2: Using Standards - 25 minutes 

In this step, participants use the EFF Worker Role Map and Standards to 
better understand the relationship between skill building and using stan- 
dards for contextualized teaching and learning. Ask participants to take a 
look at the Worker Role Map and read over the Broad Areas of Responsi- 
bility. 

Now ask participants to read through the Key Activities of “Work With- 
in the Big Picture,” which are to 

• work within organizational norms 

• respect organizational goals, performance, and structure to guide 
work activities 

• balance individual roles and needs with those of the organization 

• guide individual priorities based on industry trends, labor law/con- 
tracts, and competitive practices. 

Assign at least two participants to each of these key activities and have 
them write on Post-It Notes the skills they personally think are needed to 
accomplish each of these Key Activities. Write one skill per Post-It Note. 



For example 

— To “work within organizational norms,” a person might 
need to be able to read the personnel policies. 

— To “balance individual roles with those of the organiza- 
tion,” a person might need to be able to think critically. 



Have participants partner up for about 5 minutes and compare skills they 
identified. Once partners have had an opportunity to discuss the range of 
skills they identified, ask them to agree on three to five skills for each key 
activity. 

Now focus the group’s attention on the Standards Wheel poster with the 
16 Standards in four categories. Ask participants to place their Post-It 
Notes within the appropriate standard. This can be a fun and engaging 
activity, prompting lots of discussion and even debate. Ask participants to 
talk about what they understand from this activity in regards to the rela- 
tionship between skills and the Standards. Help them understand that 
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• To accomplish a key activity, a student might need to use more than 
one Standard; 

• Standards are much broader than skills; and 

• Standards are useful to adults at all levels of skill. 

Explain to participants that teachers can use students’ real-life roles — as 
workers and family and community members — as contexts for learning, 
as was done in this activity. The Four Purposes, the Common Activities, as 
well as the Standards, can help develop teaching and learning activities 
based on a variety of learners’ goals. 

Handout 

BREAK -10 minutes 



Step 3: Exploring Program 

Goals - 25 minutes 

In this step participants will generate a list 
of student, staff, and program goals. Make 
sure everyone has a copy of the Goals Grid 
handout. Using the Goals Grid, (see exam- 
ple below) ask team members to think 
about their personal goals for the students, 
the staff, and the program. “What do you 
hope is achieved by students, staff, and the 
program?” Have each person jot his or her 
ideas down in the appropriate column of 
their handout. 



EXAMPLE of Goals Grid 



GOALS 


Students 


Staff 


Program 


For example: 

• to pass GED 

• to upgrade employment 

• to gain new learning 
strategies 


• to make learning 
exciting 

• to create lessons 
that match different 
learning styles 


• to support learning 
that is applicable and 
meaningful in students' 
everyday lives 



2.4 
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Once members have completed their lists, ask each person to share his or 
her list of goals. As they do so, write their responses in the appropriate 
columns on the flipchart Goals Grid. If the same goals are repeated, place 
check marks by them to indicate that they have been previously stated or 
identified. 





As a group, review the list of goals. Some goals, perhaps, are closely related 
and can be synthesized or collapsed into one goal or goal category. Look 
for ways to narrow the list without leaving out any of the goals. Use the 
following question to discuss the range of organizational goals. 

“What do these goals say about the program or organiza- 
tion’s purposes?” 



Step 4: Clarifying Program Purposes - 35 minutes 
In this step you will review your organizational or program mission 
and/or vision statement. Post the flipchart sheet with these statements. 
Begin by asking a volunteer to read the mission and/or vision statement 
aloud. Ask each participant to jot down what they hear in the statements 
as organizational or program goals and purposes. Have each participant 
share his or her list. Use the following list of questions to guide a group 
discussion. 




• How do the purposes and goals stated in the mission or 
vision compare to the goals identified on the Goals Grid 
in the previous step? 

• Do we have purposes that are not stated in the vision or 
mission? 

• Are the organization or program purposes stated in other 
documents? 

• As a group can we say what our purposes are? 

The facilitator will need to make notes of the important points in the dis- 
cussion and review these points with participants. Then the group should 
come to consensus on a statement of purpose for your program. This 
statement should be written on a flipchart sheet and transferred to the 
notebook after the session. 
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BREAK -10 minutes 

Step 5: Comparing Educational and 
Program Purposes - 25 minutes 

In this step you will discuss the Four Purposes for Learning that adult 
learners identified (during the initial EFF field research) as reasons for 
returning to education: 

• To gain access to information and resources to orient themselves in 
the world 

• To give voice to ideas and opinions with the confidence that they will 
be heard 

• To take independent action to solve problems and make decisions 

• To keep learning to build a bridge to the future in a rapidly chang- 
ing world. 

Post the flipchart sheet with the EFF Four Purposes for Learning. 

Review the four EFF purposes. Use the following questions to guide a dis- 
cussion: 

• How meaningful are these purposes to our organization? Please 
explain. 

• How do these purposes compare with our students’ stated purposes? 

• How do these purposes compare with our organization’s purposes? 




Step 6: Wrapping-Up - 10 minutes 

Finish up the session with a go-around. Ask team members to comment 
on important aspects and major outcomes of the session. Wrap up by 
reviewing the goal and objectives of this session. 



Review any tasks to be completed and set dates for future meetings. 



Give each team member a copy of the handout A Purposeful 
and Transparent Approach to Education: Background and 
Case in Point # 2 and EFF Research to Practice Note 1. Ask 

them to review Note 1 and consider the Case in Point before 
the next session. 



Information to Save 
From Session One 

• Completed Goals 
Grid or List of Goals 

• Program Purpose 
Statement 
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Session Two - A Purposeful and Transparent 
Approach to Education 

Background: A quality program is one in which the organization s pur- 
poses and goals are clear. It is important for students and community 
members to understand how a program or organization relates and con- 
tributes to the student’s goals and to the broader community’s goals. 
Intake and orientation processes that are purposeful and transparent con- 
tribute to program quality by helping students to clarify what they need to 
learn in order to meet their goals. 

Educational theory and research says that learning is most effective when 
it is purposeful, when it connects to the attitudes, knowledge, skills, and 
behaviors that students need and want to fulfill their goals, and when it 
helps them become competent in their many adult roles. The purposes of 
the teaching and learning activities should be clear or transparent to the 
learner, teacher, and program. For additional background information on 
a purposeful and transparent approach to education, see EFF Research to 
Practice Note 1 in the Appendix. Administrators of an adult education 
program using EFF and wanting to improve program quality should con- 
sider the question: 

How will this agency develop a purposeful and transpar- 
ent approach to education? 

One way in which a program is purposeful and transparent is when its 
program goals are clear and understandable. In this session, the team 
members review promotional and other organizational materials and 
reflect on current intake and orientation processes to identify changes that 
would make these processes more purposeful, transparent, and useful. 
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Handout 2.5 



HANDOUT 2.5 

A Purposeful and Transparent Approach to Education: 
Background and Case in Point 2 



Background: 

A quality program is one in which the organization’s purposes and goals are clear. It is important for students 
and community members to understand how a program or organization relates and contributes to the student’s 
goals and to the broader community’s goals. Intake and orientation processes that are purposeful and transpar- 
ent contribute to program quality by helping students to clarify what they need to learn in order to meet their 
goals. 

Educational theory and research says that learning is most effective when it is purposeful, when it connects to 
the attitudes, knowledge, skills, and behaviors that students need and want to fulfill their goals, and when it 
helps them become competent in their many adult roles. The purposes of the teaching and learning activities 
should be clear or transparent to the learner, teacher, and program. For additional background information on a 
purposeful and transparent approach to education, see EFF Research to Practice Note 1 in the Appendix. 
Administrators of an adult education program using EFF and wanting to improve program quality should consid- 
er the question: 

How will this agency develop a purposeful and transparent approach to education? 

One way in which a program is purposeful and transparent is when its program goals are clear and understand- 
able. In this session, the team members review promotional and other organizational materials and reflect on 
current intake and orientation processes to identify changes that would make these processes more purposeful, 
transparent, and useful. 



Case in Point: 

An eleven-year-old community based organization, The Pierce CommunityBuild Adult Education Cen- 
ter serves mostly young (18 to 25), low-income adults. The program offers students an opportunity to 
obtain a GED, learn a building trade, and gain valuable life skills such as budgeting and civics. Most of the 
students come to the program to get their GED, and they become frustrated when instruction is not tied 
directly to GED preparation. 

The program has decided that it needs to present a clearer picture to potential students about the pro- 
gram’s purposes and that it has to get a clearer picture from students about their learning purposes and 
their life goals. In fact, all the processes connected to the teaching and learning (intake, placement testing, 



Session 2 Goals: 

To review the organization’s intake and orientation processes and identify 
changes that would enhance a purposeful and transparent approach to 
education. 

Objectives: 

By the end of this activity, participants will be able to 

• clarify the purposes of the educational programs 

• identify specific steps for making the program purposeful and 
transparent 

• understand the organization’s current intake process. 
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Materials Required: 

• Session # 2 handout Background and Case 
in Point (2.3) 

• Organizational brochures, recruitment 
materials, newsletters, other publications 

• Flipchart sheets labeled Changes to Make, 
Intake Process, and Orientation Process 



Facilitator’s Note: This session 
focuses on the purposefulness and 
transparency of organizational goals 
and processes. In this activity you will 
examine the organization’s intake 
and orientation process as a way of 
better understanding the concept of 
a purposeful and transparent 
approach to education. 



▼ 



Time Required: 2 1/2 hours 



Steps in Session Two 



Settle-In and Session Overview - 20 minutes 
Remind team members of the overall project goals, the outcomes from the 
previous session, and the goals and objectives of this session. Involve par- 
ticipants in a discussion on what makes something purposeful and trans- 
parent (you may want to refer to EFF Research to Practice Note 1 — handout 
2.6 — on purposeful and transparent approaches to education and to the 
handout Case in Point #2). 

Step 1: Reviewing Promotional Materials - 40 minutes 
In this step you will examine your organization’s promotional materials — 
brochures, recruitment materials, newsletters, other publications, etc. If 
there are six or more people involved in the meeting, you can have mem- 
bers work in twos or threes to examine different materials. Or they can 
work as a whole group to examine a specific promotional piece. For this 
step, use these guiding questions: 

• Do our recruitment and other program materials reflect 
our purpose? 

• What would make our purposes more transparent? 

Distribute promotional materials, etc., to group members. Ask partici- 
pants to scan materials for content that makes clear the program’s purpose 
and/or goals. The content can be highlighted or underlined for quick ref- 
erence. Give participants about 10 minutes to accomplish this task. Then 



AN EFF HANDBOOK FOR PROGRAM IMPROVEMENT 

31 




Phase 2: Examining Your Program - Session Two 



r\ 



go around and have each team share examples of content that reflect the 
program’s purposes. They might also share what changes they feel would 
make the program purposes transparent or clearer. Record 
the key points in response to these questions on flipchart 
paper. Make sure that everyone has an opportunity to 
share his or her views and opinions. 



Next, brainstorm a list of changes or improvements in 
recruitment that would make the organizations purposes 
clearer. Document brainstorm responses on “Changes to 
Make” flipchart. 



Step 2: Reviewing Intake Process - 40 minutes 
In this step you will examine the organization’s student 
intake process. Ask team members to think about the orga- 
nization’s student intake process. As a group, list the steps 
involved in the organization’s current intake process, for 
example, collecting information from new students or 
identifying students’ purposes for attending the program. 



Document changes in a 
“Changes to Make” list. 



Changes to Make in 
Recruitment 



Document in “Intake 
Process” list. 



After the group completes the list of steps in the current 
intake process, ask the group to answer the following ques- 
tions. Facilitator will record responses on a flipchart sheet. 

During the intake process, when does the learner 
have an opportunity to understand about the pro- 
gram or the program’s purpose is? 



Intake Process 



During the intake process, when does the program 
have an opportunity to understand about the learn- 
er (learner’s goals)? 



Break -10 minutes 
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Step 3: Reviewing Orientation Process - 40 minutes 
In this step you will examine your organization’s orientation process. Ask 
team members to think about the student orientation process. Make a list 
of what currently happens during the orientation process. After the list is 
completed, ask the group to answer the following ques- 
tions and document responses on a flipchart. 

• During the orientation process, when does the learn- 
er have an opportunity to understand the program 
or the program’s purpose? 



• During the orientation process, how do you current- 
ly find out what learners’ goals are? 

• During the orientation process, how do you help 
learners’ go deeper into why they have certain goals? 
For example, “You want to get a GED in order to . . 

yy 



• During the orientation process, how do you help learners make 
informed decisions about what they need to learn in order to achieve 
their goals? 



Document in 
an “Orientation Process” list. 




Use the following flipchart grid to capture suggested changes for both the 
intake and orientation processes. All of the notes will need to be typed up 
and filed after the meeting for future reference. 




INTAKE/ORIENTATION GRID 



Intake Process 



CHANGES TO MAKE 



Orientation process 




Finish this step by summarizing the notes on the flipchart page above. 
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Step 4: Wrapping-Up -10 minutes 

Finish up the session with a go-around. Ask team members to comment 
on what was most important about the learning activity. 



Wrap up by reviewing the goals and objectives. Ask team members to 
comment on the overall process. Review any tasks to be completed and 
remind participants of dates for future sessions. 

Pass out the handouts A Contextualized Approach to Cur- 
riculum and Instruction: Background and Case in Point 3 
and Research to Practice Notes 2 and 3 and ask the team 
members to read before the next session. 



Information to Save 
From Session Two 

• List of “Changes to 
Make” in recruitment 

• List of “Changes to 
Make” in Intake and 
Orientation Process 
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Session Three - Building Expertise 
Using a Contextualized Approach to 
Curriculum and Instruction 

Background: In effective adult teaching and learning environments, 
research shows that attention to context — the environments, conditions, 
and circumstances in which learners exist — helps students apply knowl- 
edge, skills and processes across contexts, and that the learning context 
itself contributes to the learning process. For additional background infor- 
mation, see EFF Research to Practice Note 3. Understanding learners’ con- 
texts is important for identifying EFF Standards that would help learners 
achieve their goals. It is also important to understand the many contexts of 
learners’ lives when focusing and designing the learning activity. Learning 
activities that directly connect to learners’ lives are more effective. 

Building expertise is a complex developmental process in which new 
knowledge is built on prior knowledge. Students need cognitve and 
metacognitive strategies to learn and apply new information. For addi- 
tional information on building expertise, see EFF Research to Practice Note 
2. The EFF framework, particularly the EFF Standards and the 
Teaching/Learning Cycle, gives teachers a structure for designing learning 
activities that address the issues arising from the contexts of students’ real 
lives while also building strategies and basic skills. 

Administrators can support contextualized teaching and learning by clari- 
fying both their programs and students’ contexts. Awareness of these con- 
texts might mean that administrators foster alignment of program goals 
with student goals and student contexts, budget for materials relevant to 
the teaching and learning contexts of students and the program, plan and 
support staff development activities that focus on contextualized curricu- 
lum development, and design and monitor assessment and evaluation 
processes based on contextualized instruction. 

Administrators can play an important role in supporting activities that 
extend learning to different contexts outside of the program. To do this, it 
might mean that administrators also network with and disseminate pro- 
gram goals and activities to community partners such as employers, other 
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educational institutions, community-based organizations, and social ser- 
vice agencies. 

The question for the team: 

How will this agency help students build expertise 
through a contextualized approach to curriculum and 
instruction? 

Handout 2.7 



HANDOUT 2.7 

A Contextualized Approach to Curriculum and Instruction: 
Background and Case in Point 3 

Background: 

in effective adult teaching and learning environments, research shows that attention to context — the 
environments, conditions, and circumstances in which learners exist — helps students apply knowledge, skills 
and processes across contexts, and that the learning context itself contributes to the learning process. For 
additional background information, see EFF Research to Practice Note 3. Understanding learners’ contexts is 
important for identifying EFF Standards that would help learners achieve their goals. It is also important to 
understand the many contexts of learners’ lives when focusing and designing the learning activity. Learning 
activities that directly connect to learners’ lives are more effective. 

Building expertise is a complex developmental process in which new knowledge is built on prior knowledge. 
Students need cognitve and metacognitive strategies to learn and apply new information. For additional 
information on building expertise, see EFF Research to Practice Note 2. The EFF framework, particularly the 
EFF Standards and the Teaching/Learning Cycle, gives teachers a structure for designing learning activities 
that address the issues arising from the contexts of students’ real lives while also building strategies and 
basic skills. 

Administrators can support contextualized teaching and learning by clarifying both their programs and 
students’ contexts. Awareness of these contexts might mean that administrators foster alignment of program 
goals with student goals and student contexts, budget for materials relevant to the teaching and learning 
contexts of students and the program, plan and support staff development activities that focus on 
contextualized curriculum development, and design and monitor assessment and evaluation processes 
based on contextualized instruction. 

Administrators can play an important role in supporting activities that extend learning to different contexts 
outside of the program. To do this, it might mean that administrators also network with and disseminate 
program goals and activities to community partners such as employers, other educational institutions, 
community-based organizations, and social service agencies. 

The question for the team: 

How will this agency help students build expertise through a contextualized approach 
to curriculum and instruction? 



Case in Point: 

Kenesshia, one of Ms. Franklin’s GED students, has recently moved into her own apartment. She has 
been working on math skills in preparation for the GED test. Ms. Franklin is interested in knowing how 
Kenesshia is applying the math she is studying in class to her new situation, managing her own apart- 
ment and finances. One day Ms. Franklin asked Kenesshia how things were going with the apartment. 



Session 3 Goals: 

To better understand the relationship between skill building, EFF Stan- 
dards, and contextualized teaching and learning. 
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Objectives: 

By the end of this session, participants will be 

able to 

• identify a variety of learners’ real-life contexts 

• understand common barriers to contextual- 
ized approaches to curriculum and instruction 

• understand how the EFF Standards can be 
used to connect learners’ contexts to specific 
skills to be learned 

• articulate new questions about the various 
roles of administrators and how they might 
support contextualized approaches to teaching 
and learning 

• identify program improvements that might 
help the organization take a more contextual- 
ized approach to curriculum and instruction. 

Materials Required: 

• EFF Research to Practice Notes 2 & 3 (handouts 2.8 and 2.9) 

• Constructivist Learning handout (2.10) 

• Teaching/Learning Cycle handouts (2.11) 

• Teaching/Learning Cycle Grid handout (2.12) 

• Flipchart sheet labeled Students’ Real-Life Contexts 

• Contextualized Approach to Curriculum and Instruction grid on 
flipchart 

• Teaching and Learning Cycle grid on flipchart 

• EFF Content Standards Book 

Time Required: 3 hours 



▼ 

Facilitator’s Note: It is important for 
teachers and program administrators to 
understand not only the real-life contexts 
of students but also how program and 
organizational contexts matter in teach- 
ing and learning. Program administrators 
need to be aware of the variety of ways 
in which “contexts” and contextualized 
teaching apply to quality programs and 
quality instruction. This session guides 
the program improvement team 
members through a reflective process 
to examine how student contexts impact 
instruction, review some common 
barriers to contextualized instruction, 
and examine two examples of 
contextualized instruction. 
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Steps in Session Three 



Settle-In and Session Overview -10 minutes 
Review goals and objectives for the session. Remind participants of the 
overall project goals, what happened in previous sessions, and summarize 
where the project is currently. 




Step 1: Identifying Learning Context - 20 minutes 
In this step, participants will identify various contexts of learners’ lives. As 
a group, brainstorm a list of issues from the contexts of learner lives that 
might frame instruction. 

What are real-life contexts you’ve heard 
students talk about? 

Record the list on flipchart. 

Get a discussion going on the program’s experiences 
designing lessons that incorporate students’ real-life 
contexts. Use the following questions as discussion 
prompts. 



Sample List 

Real-life contexts that 
students talk about: 

• finding a living wage job 

• keeping children safe in the 
neighborhood 

• losing welfare benefits 

• dealing with mother's death 

• rising costs of living 
expenses 



• What are the program’s experiences with using real-life contexts to 
frame instruction? 



• What was the motivation for taking this approach? 



• What was learned from these experiences? 



• What might get in the way of doing contextualized learning? 




Record the group’s responses to these questions on flipchart. Keep for 
future reference. 
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Step 2: Understanding Contextualized Teaching and 
Learning - 50 minutes 

In this step participants will work in pairs to examine two examples of 
contextualized approaches to curriculum and instruction. These examples 
are excerpts from EFF Research and Practice Note 3, the section “Putting 
Contextualized Learning to Work in Your Program,” written by Marilyn 
Gillespie. The examples demonstrate how a program might implement 
the program practice of a contextualized approach to curriculum and 
instruction. Make sure participants have copies of the program examples 
from the Research to Practice Note 3, on contextualized instruction. 
Assign an example to each pair. 

Have pairs read and discuss: 

Example 1: Teachers and students use the EFF framework to construct 
learning opportunities that focus on the development and 
practice of skills students need to carry out activities and 
accomplish purposes in the lives. 

OR 

Example 2: Teachers use the EFF Framework to integrate “found lessons” 
that arise from in-class or out-of-class student needs into an 
overall plan. 

Using the EFF Content Standards book ask participants to review and 
become familiar with the Use Math to Solve Problems and Communicate, 
Speak So Others Can Understand, Convey Ideas in Writing, Learn Through 
Research, and Solve Problems and Make Decisions Standards. 

Focus the group’s attention on the bulleted items under each standard, 
which are the components of the Standard. Explain that the components 
describe what the Standard means and that you use the Standard to do 
something. 

Have each pair discuss their example, using the following guiding ques- 
tions: 

• Why do you think the teacher used these particular Standards? 

• What other Standards might the teacher use? 
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Now, still in pairs, think about your program contexts. Answer the follow- 
ing on the Contextualized Approaches to Curriculum and Instruction 
form. 

• Are there program contexts, issues, and concerns that might 
be used for curriculum development? 

• Are there community contexts, issues, and concerns that 
might be used for curriculum development? 

• What program improvements could help the organization 
take a more contextualized approach to teaching and learn- 
ing? Make a list. 

• As an administrator, what questions do you have about your 
role supporting contextualized approaches to curriculum 
and instruction in your program or organization? 

Have each pair share with the group what they discussed. Use the flipchart 
to record the suggested program issues, improvements, and questions as 
in the example below. 




Contextualized Approaches to Curriculum and Instruction 


Program and Community Issues 


Program improvements for 
Contextualized Approaches 


For Example: 

The lack of awareness about 
community based recreational 
activities for children. 


For Example: 

Identify and use newsletters from 
local CBOs for lessons that are 
relative to the issues. 



Questions 

For Example: 

• What are information sources for identifying local CBOs, their mission and 
program activities? 

• How do I build effective partnerships for receiving and disseminating 
information? 



Break -10 minutes 
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Step 3: Understanding Constructivist Teaching 
and Learning - 30 minutes 

Explain to participants that they just experienced an important step in 
building expertise or “constructivist” learning, reflecting on what they 
already know about contextualized learning in their organization and 
then adding new information. Use the following Constructivist Learning 
handout to guide a discussion on the approach. 

Handout 2.10 



HANDOUT 2.10 

Constructivist Learning 

Building Expertise 

Constructivism, in educational theory and practice, means that learning is a process of constructing new 
meanings in which prior knowledge and experiences are the basic building blocks for building expertise. 
Constructivism is a theory of learning and of knowing. For additional background information, see EFF 
Research to Practice Note 2 in the Appendix. From conception, the human brain accumulates and stores 
knowledge, information, and skills which become the basic building blocks for understanding and mak- 
ing sense of the world. As knowledge and experience are added, a person gains new understanding and 
perspective. A teacher who approaches teaching and learning from a constructivist standpoint starts by 
finding out what learners already know and have experienced that relates to a topic or idea and helps the 
students build on that knowledge and experience to gain new concepts, information, and skills. The 
teaching and learning process is interactive and dynamic. Review the chart below which illustrates a con- 
structivist learning process. 



Constructivist Learning Process 



Student’s prior knowledge, 




New Information 






skills, background, and 




and Experiences 




New 


experience 




• Reflections 




Meanings 


• Language 


+ 


• Information 


— 


• Social Skills 




• New Skills 




and 


• Phonemic Awareness 




• Knowledge 




• Sense of Community 

• Historical Perspective 




• Analysis 

• Practices 




Knowledge 



Constructivist teaching and learning is a 

process of discovery. It is one in which teachers 
and students use the EFF framework to identify, 
reflect on, and revise their own mental models of 
adult role performance. For example, the EFF 
Standards show the various components of a 



particular skill. The students and teacher can review 
those components of performance and decide what’s 
already known about the skill and what needs to be 
learned or improved. This review and reflection 
process is the first step in constructivist teaching 
and learning. 




Now ask participants to think about think about: 

In what ways do teaching and learning in our program 
enable students to build expertise ? 

Get a discussion going for about 15 minutes. The facilitator/scribe should 
take note of the comments on a flipchart sheet to be used in final plan- 
ning. 
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Step 4: Reviewing the Teaching and Learning Cycle - 60 minutes 
In this step, participants will examine the Teaching and Learning Cycle. 
They will look at the steps in the cycle and determine what already hap- 
pens in their programs that is relative to the teaching and learning cycle 
and what program changes might better support the activities of the 
teaching and learning cycle. 

Distribute the EFF Teaching and Learning Cycle handout, with the eight 
steps, guiding questions, and sample tools. Briefly review each of the eight 
steps and the guiding questions. Have the team members work in dyads or 
triads. 



Handout 2.11 




• Discuss each step of the cycle and how these steps help students build 
expertise. 

• Discuss each of the steps in the cycle and decide if and when the 
activities outlined in the teaching and learning cycle happen in your 
program 

• Discuss what program improvements or changes would better sup- 
port the teaching and learning activities. 
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The small groups should work for 30 minutes using the Teaching and 
Learning Cycle Grid to record their comments. 

Handout 2.12 



HANDOUT 2.12 

EFF Teaching and Learning Cycle Grid 



EFF Teaching/Learning 
Cycle Steps 


Implications for 
Administrators 


Changes or 

Program Improvements 


1. Determine individual learner’s 
goals and purposes and 
identify the Standards that 
will help him/her achieve 
them. Identify student’s prior 
knowledge about these 
goals and Standards. 


For example: 

Students complete an 
interest survey. 


For example: 

Discuss student goals 
at intake and share with 
teachers who will use to 
plan with students. 


2. In a group identify a shared 
interest, purpose or goal and 
determine the group’s prior 
knowledge of this topic. 
Identify the Standard that will 
help the group address this 
shared goal. Make clear the 
connection between the class 
focus and individuals’ needs. 






3. Design a learning activity to 
address the real-life concerns 
of the learner(s). 






4. Develop a plan to capture 
evidence and report learning. 






5. Carry out the learning activity. 






6. Observe and document 
evidence of performance 
of the Standard. 






7. Evaluate and reflect on 
how what was learned is 
transferable to real-life 
situations? 






8. Determine next steps to help 
learners meet their goals. 
(Return to Step 1 and/or 2) 








The facilitator then brings the small groups back together to share what 
they discussed. 

The facilitator should record responses from each group in the appropri- 
ate column of the flipchart sheet, for each step in the teaching and learn- 
ing process: 

1. What happened in each step? 

2. What program improvements or changes would better 
support the teaching and learning cycle? 
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For example, for Teaching/Learning Cycle Step 1 , the program may have 
students complete an interest survey at intake. A possible program 
improvement might be to have a more explicit goal-setting discussion at 
intake and then revisit student goals as part of lesson planning. 

Tell this team that EFF has developed an on-line “toolkit” to help teachers 
use the Teaching/Learning Cycle that is found at: 

http://cls.coe.utk.edu/efftlc/ 



Step 5: Wrapping-Up -15 minutes 

Wrap up this session by restating the goals and objectives as well as the 
steps in this session: 

1 . The team identified a variety of learners’ real-life contexts for which 
curricula might be designed, 

2. The team discussed common barriers to contextualized instruction, 

3. The team examined the teaching and learning cycle to determine what 
they do and/or know about the steps in the cycle, 

4. The team listed what administrators might do to support teaching and 
learning cycle activities, 

5. The team brainstormed program improvements 
and changes that support the implementation of 
teaching and learning cycle activities. 

Check in with participants to see if they feel that the 
session’s goals and objectives were met. Invite partici- 
pants to share comments and reactions to the session. 

Review any tasks to be completed and remind team 
members of dates for future sessions. Pass out copies of 
Joan Benz’s article on assessment and Program Goals 
Grid of List of Program Goals from Session One, and 
Assessment, Accountability and Program Improve- 
ment: Background and Case in Point 4 handouts to 
read before the next session. 



Information to Save From 

Session Three 

• List of Barriers to 
Contextualized Teaching 
and Learning 

• Examples of Contextualized 
Approaches to Curriculum 
and Instruction 

• List of Current Teaching and 
Learning Approaches that 
enable Students to Build 
Expertise 

• List of Program Improvements 
or Changes That Would 
Support the Teaching and 
Learning Cycle 

• List of Administrative 
Activities That Would Support 
the Teaching and Learning 
Cycle 
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Session Four - Assessment, Accountability 
and Program Improvement 

Background. In a quality program, assessment and instruction are 
aligned. Administrators need to know how student assessment is currently 
done and what needs to happen to move towards an approach that is 
grounded in cognitive science research on the development of expertise. 
In order to support the alignment of instruction and assessment, the qual- 
ity improvement team reviews current assessment practices and compares 
them with assessment practices based on the EFF Framework. The team 
also reviews what and how the program collects data and documents stu- 
dent progress in order to identify changes that would improve organiza- 
tional performance and outcomes. 

The question for the team is: 

How will this agency align assessment with student and 
program goals and document student progress and 
change? 



Handout 2.14 
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Session 4 Goals: 

To explore the organization’s assessment practices and consider how 
assessment might look if it were based on the EFF Assessment Frame- 
work; to review the organization’s current documentation methods and 
accountability processes. 

Objectives: 

By the end of the session, participants will be able to 

• identify the organization’s current assessment processes 

• consider what an assessment process would look like based on the 
EFF Assessment Framework 

• identify the organization’s current documentation methods and 
processes 

• understand what the program is accountable for, to whom, and for 
what 

• list program activities that would lead to a more systematic approach 
to program improvement. 

Materials Required: 

• Questions on Assessment handout (2.15) 

• EFF vs. Traditional Assessment handout (2.16) 

• Current Approach to Assessment Grid handout 

• Joan Benz’s article on assessment in the EFF 
classroom (2.13) 

• Changes to Make flipchart sheet 

• Targe table for sorting documents 

• Goals list from Session 1 - on flipchart and 
copies to hand out 

• Documentation Methods on flipchart or 
board 

• Flipchart sheets labeled Current Approach To 
Assessment Grid and Responses to Joan 
Benz’s Article 

Time Required: 3 hours 



(2.17) 

~w 

Facilitator’s Note: Review examples 
of Traditional versus EFF Approaches to 
Assessment handout in the Appendix. 

It will be useful to have the list of 
program goals identified in Session 1. 
The facilitator should gather and bring 
to the meeting various forms of program 
documentation and encourage other 
group members to do the same. Docu- 
mentation includes student information 
forms, test results, portfolios, copies of 
program completion certificates, state 
reporting forms, etc. Before the session 
prepare a flipchart sheet with the 
information across the top as in the 
example on page 50. 
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Steps in Session Four 

Settle-In and Session Overview 

Explain that this session focuses on assessment and documentation. 
Explain the goal and objectives of the session. 



Step 1: Discussing Current Approaches 
to Assessment - 60 minutes 

In this step participants will use questions to reflect on the program’s cur- 
rent approaches to assessment. Make sure that participants have copies of 
the Current Approach to Assessment handout, which list the questions 
below. 

• What kinds of learning are measured? (skills, competencies, etc.) 

• How is learning measured? 

• What evidence is gathered to evaluate performance? 

• How is this evidence used? 

• Is the learning scored? If so, how? 

• What is the student’s role in the assessment process? 

• How does assessment connect Handout 2.15 
to instruction? 

• How does assessment connect 

to curriculum and content 

standards? 

• How does assessment connect to educa- 
tion theory and research? 

• How are assessment results 
documented? 

Ask the group to think about ways in which 
the program approaches assessment, i.e. 
determines students’ knowledge when they 
enter the program, what students are learn- 
ing in the program, and how students’ 
involvement in the program leads to new 
changes in their roles as adults. Then have 



HANDOUT 2.15 

Questions on Assessment 

What kinds of learning are measured? (skills, competencies, etc) 
How is learning measured? 

What evidence is gathered to evaluate performance? 

How is this evidence used? 

Is the learning scored? If so, how? 

What is the student’s role in the assessment process? 

How does assessment connect to instruction? 

How does assessment connect to curriculum and content standards? 
How does assessment connect to education theory and research? 
How are assessment results documented? 
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participants respond in writing to as many questions as they can in the 
“current approach” column. 



Handout 2.16 



HANDOUT 2.16 

Current Approach to Assessment 



Questions 


Current Approach 


What kinds of learning 
are measured? (i.e. skills, 
competencies, etc.) 




How is learning measured? 




What evidence is gathered 
to evaluate performance? 




How is this evidence used? 




Is the learning scored? 
If so, how? 




What is the student’s 
role in the assessment 
process? 




How does assessment 
connect to instruction? 




How does assessment 
connect to curriculum and 
content standards? 





Post the prepared flipchart sheet of the Current Approach to Assessment 
grid and discuss each question and invite participants to contribute their 
ideas. The facilitator should make note of the group’s responses, under- 
score questions the group found difficult to respond to, and any new 
questions raised. 

Next, participants will discuss and compare their current approach to 
assessment to the EFF approach to assessment. Distribute copies of the 
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Traditional versus EFF Approaches to Assessment with the EFF column 
filled in. Ask participants to review the list and compare how they current- 
ly approach assessment with the EFF approach to assessment. 



Handout 2.17 



HANDOUT 2.1 7 

Traditional Versus EFF Approaches to Assessment 





Current Approach 


EFF Approach ' 


Construct (what is 
measured) 


Discrete knowledge, skills, or 
competencies 


Integrated knowledge, skills, 
strategies, and abilities 


Task format 
(how it is measured) 


Questions or problems posed in 
isolation or with a small amount of 
supporting context, tasks (test items) 
usually of short duration, and typically 
require recall and analysis 


Task situated in activity that closely 
mirrors real-world activity, may take 
place over an extended period of time, 
and can involve creativity, strategic 
thinking, and problem-solving 


Response format 
(evidence used to 
evaluate performance) 


Selected response or short, written 
response, usually collected at one time 


Constructed response, oral, written, 
graphic, and/or behavioral response or 
performance that may be collected 
over a period of time (as in a portfolio) 


Scoring format 


Scored as right or wrong or on a 
one-dimensional scale (assigned a 
letter or number grade) 


Scored with a rubric that identifies 
dimensions of performance and 
qualitative differences in levels of 
performance 


Role of student 


Student are not told what the questions 
will be before the test and typically do 
not assess their own performance 


Students know in advance the nature 
of the task and how their performance 
on the task will be evaluated and can 
self-assess and monitor their own 
performance 


Connection to 
instruction 


Separate from instruction but tests 
discrete skills or knowledge that are 
the objects of instruction 


Embedded in instructional and 
learning activities or on-demand tasks 
that have the same structure as 
instructional and learning activities 


Connection to 
curriculum and 
content standards 


Assessed knowledge, skills, and 
abilities may be poorly or incompletely 
aligned with curriculum and content 
standards 


Assessed knowledge, skills, and abili- 
ties are aligned with standards and 
standards-based curricular content 


Connection to theory 
and research 


Content and structure of assessments 
may be related to skill hierarchies but 
are typically not derived from a theory 
of learning, development, or expert 
performance 


Content and structure of the 
assessments are derived from 
analyses of adult learner 
performance, cognitive science 
models of developing expertise, 
and adult learning theory 



Step 2: Reflecting on Assessment in an 
EFF Classroom - 40 minutes 

Check in with participants to make sure that they have copies of the Joan 
Benz article, “Assessment in the EFF Classroom,” distributed at the last 
session. Ask participants to scan and recall key points. Use the following 
questions to guide a discussion. 

• What do you find most interesting about the article? 

• What stands out for you when comparing your program’s 
current assessment approaches to the EFF assessment 
approaches? 
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The facilitator should make note of the responses to these questions on a 
flipchart sheet. Add any ideas about possible changes in your programs 
assessment processes to the Changes to Make sheet. 




Break -10 minutes 



( The following activity is adapted from "How Are We Doing?" NCSALL 
Teaching and Training Materials) 

Step 3: Reviewing Documentation Methods - 30 minutes 
In this step, participants move from examining assessment to considering 
the organization’s current documentation methods, what data they col- 
lect, and how the data are used. The group will start by sorting the differ- 
ent kinds of documents into categories on a large table. For example, they 
may have a category with student registration forms, GED Practice Test 
results, student journals, etc. Documents that don’t fit easily into a catego- 
ry can become a new category. 

Now, have group members name the different categories and list them on 
the flipchart sheet under the Documentation Method column. Explain to 
the group that there are several things to consider about documentation: 

• Documentation by whom? Who is responsible for collect- 
ing the document? 

• Documentation for whom? Who uses the information on 
the form? 

• How often is the documentation collected? 



Documentation 

Method 


By whom 


For whom 


How often 


How used 


Information 

Documented 


Registration Form 


Program 

aide 


Program 


Every 

program cycle 


For 

demographics 


• Basic demographics 

• Educational background 


Placement test 












Attendance form 












Individual learning 
plan 












Observation 
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• How is the documentation used? 

• What information is documented? 

Now ask the group members to consider each of the documentation 
methods and complete the grid together, as a group. 



Step 4: Matching Program Goals and 

Documentation Methods -20 minutes 
Distribute copies of the “Goals List” generated in the first session. Post a 
large version of the list on the flipchart stand as well. Now ask participants 
to work in pairs and use the following questions to compare the list of 
goals with the documentation grid: 

• For which goals does the program collect documentation 
of achievement? 

• For which goals does the program collect data that tells 
when the goals are met? 

• For which goals does the program not collect data? No 
documentation? 

Invite each pair to share a goal and the method of documentation. The 
facilitator should make note of the goals documented and the methods 
used. There may also be goals not documented or documentation that 
matches no goals. 





Get a discussion going about the results of this activity. The 
facilitator should make notes of any new questions and con- 
cerns. Add suggested changes to Changes to Make sheet. 



Step 5: Wrapping-Up -10 minutes 

Wrap up this activity by restating the goals and objectives. Check 
in with participants to see if they feel that the goals and objec- 
tives were met. Invite participants to share comments and reac- 
tions to the activity. Review any tasks to be completed and set 
date for the next session. Give participants copies of the article 
by Juliet Merrifield and Accountability, Change and Quality: 
Background and Case in Point 5 to read before the next session. 



Information to Save 

From Session Four 

• Notes From 
Discussion on 
Comparing 
Program’s Current 
Assessment 
Approaches to 
EFF Assessment 
Approaches 

• Documentation 
Methods Grid 

• Notes on Goals 
Not Documented 
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Session Five - Program Improvement: 
Accountability, Change and Quality 

(Adapted from "How Are We Doing ? " NCSALL Teaching and Training 
Materials) 

Background: A systematic approach to continuous program improve- 
ment means having clear organizational goals and purposes that align 
with student goals and national policies. It also means having a process for 
documenting and monitoring not only student progress and outcomes 
but every aspect of the organization’s performance. Performance account- 
ability, an important aspect of program improvement, is a way of convey- 
ing to stakeholders “what” and “how” an organization is doing. 

In this session, the quality improvement team builds expertise through a 
process that examines how organizational changes have occurred in the 
past, what changes would lead to a higher quality program in the future, 
and identifies changes that would improve organizational performance 
and outcomes. 

A quality improvement program, such as the Baldrige National Quality 
Program, offers a framework for approaching quality improvement in an 
organization. Improvement processes require organizations to have a sys- 
tem for collecting and analyzing data on student progress and using it to 
review the effectiveness of program components and revise program goals. 

The team might ask: 

How will this agency take a systematic approach to 
program improvement? 

Here’s another case in point. 
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Handout 2.19 



HANDOUT 2.19 

Accountability, Change and Quality: 
Background and Case in Point 5 



(Adapted from "How Are We Doing?" NCSALL Teaching and Training Materials) 

Background: 

A systematic approach to continuous program improvement means having clear organizational goals and pur- 
poses that align with student goals and national policies. It also means having a process for documenting and 
monitoring not only student progress and outcomes but every aspect of the organization’s performance. Perfor- 
mance accountability, an important aspect of program improvement, is a way of conveying to stakeholders 
“what” and “how” an organization is doing. 

In this session, the quality improvement team builds expertise through a process that examines how organiza- 
tional changes have occurred in the past, what changes would lead to a higher quality program in the future, and 
identifies changes that would improve organizational performance and outcomes. 

A quality improvement program, such as the Baldrige National Quality Program, offers a framework for approach- 
ing quality improvement in an organization. Improvement processes require organizations to have a system for 
collecting and analyzing data on student progress and using it to review the effectiveness of program components 
and revise program goals. 

The team might ask: How will this agency take a systematic approach to program improvement? 



Case in Point: 

Mr. Johns, the education coordinator at the CommunityBuild Adult Education Center, returned from the 
state’s annual adult basic education conference excited about the possibility of receiving additional funds 
for the Center’s programs. At the conference, he learned about the state’s new accountability require- 
ments. To meet the accountability requirements and access the new funds, the Center needs to collect 
more information on the progress of students. 

Mr. Johns spoke to Ms. Franklin about the new requirements, and she flatly stated, “I just don’t have time 
to do more paperwork if I’m to really do my job as a teacher.” Mr. Johns understood Ms. Franklin’s 
dilemma. Mr. Johns then spoke with the Center’s Director about the possibility of getting new funds for 
the adult education program. The director wanted to know how the new accountability requirements 
help to meet the needs of the students. 

Mr. Johns was a bit overwhelmed by what he was hearing and decided to raise the issue in the next staff 
meeting. At that meeting, all of the staff agreed that the organization needed to be strategic in making 
program changes and that the entire organization needed to determine how to strengthen program 
processes and outcomes. One staff person summed it up this way, “One person can’t do it alone; it takes 
the whole staff and anyway, it’s all about helping students to meet their goals.” 



Session 5 Goals: 

To build on the organization’s experiences with implementing changes so 
as to better understand how to proceed with program improvement and 
accountability. 

Objectives: 

By the end of the session, participants will be able to 

• reflect on current program practices for implementing change 

• have a better understanding of performance accountability in adult 
education 

• list program changes that would lead to a more systematic approach 
to program improvement. 
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Materials Required: 

• Questions About Change handout (2.20) 

• Flipchart pages from Steps 1 and 2 

• Program goals list generated in Session 1 

• Merrifield’s article on accountability (2.18) 

• Documentation from program Changes to 
Make forms from previous sessions 

• EFF and Baldrige matrix drawn on flipchart 
sheet (2.21) 

Time Required: 2 hours and 45 minutes. 



▼ 

Facilitator’s Note: This session 
requires using the list of goals identified 
in Session 1 . 

Most of the steps in this session require 
participants to read articles and other 
materials. Check in to make sure 
participants read the Merrifield article on 
performance accountability before the 
session begins. If participants have not 
read the article beforehand, then make 
adjustments to the activity. Perhaps, 
in small groups, they can each read a 
particular section. 

The facilitator will also need to pull 
together all the documentation of 
program changes generated during the 
previous sessions. This information will 
be useful for prioritizing and planning 
program changes. 



Steps in Session Five 

Settle-In and Session Overview -10 minutes 
Review the goals and objectives for the session. Remind participants that 
this is the last session in this phase to assess and reflect on current pro- 
gram issues and practices. The next phases are to plan, implement, docu- 
ment and evaluate program improvement changes. 



Step 1: Reflecting on Prior Program Changes - 45 minutes 
In this step, participants examine how program improvements or change 
currently happens in their organization. Ask participants to think about a 
recent change effort at work in which they were personally involved. Each 
person should think of one experience with change and write responses to 
the following list of questions. 
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Handout 2.20 




Have participants pair up, share, and discuss responses to the questions 
and consider if 

• the change was part of or an entire system- wide change 

• the change was resisted and by whom 

• the change process was smooth or rough 

• the change process resulted in the desired outcome 

• leading the change was a comfortable or a not so comfortable role. 

Give participants at least 15 minutes to discuss, in pairs, before having 
each pair report out to the larger group. Pairs should make a 5-minute 
presentation to the group, sharing what was most important about their 
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discussion, the lessons learned, and the new 
questions they might have about change. The 
facilitator and scribe should record this informa- 
tion on a flipchart sheet. 



Break -10 minutes 



Step 2: Reflecting on Performance 

Accountability - 40 minutes 
In this step the group will discuss the article on 
performance accountability written by Juliet 
Merrifield, who is a leading researcher in the 
field of adult education. Begin by asking each 
participant’s reaction to the article. 



Important points about change 



Lessons learned 



New questions 




After hearing everyone’s comments, post a flipchart sheet with the follow- 
ing quotes. 

Quotes 



“I would like to suggest that developing performance account- 
ability is not just technically challenging but also challenges 
our values.” 



“ What is counted becomes what counts.” 



“Two kinds of capacity — to perform and to be accountable — 
are linked.” 



“In [programs], mutual accountability would engage members 
of the organization in creating a common vision, determining 
goals and customer expectations, and designing effective 
means of monitoring processes and results.” 

Discuss each of the quotes. Have participants interpret and share their 
reactions to each quote. Remember that Merrifield outlines four impor- 
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tant topics in considering accountability: 

1. agreement on performance 

2. capacity to be accountable 

3. tools for measuring accountability, and 

4. mutual accountability 

During the discussion, the facilitator should encourage participants to 
focus on and get clear about (1) the implications of these four topics to 
their overall program and (2) what each of the quotes might mean in 
terms of their overall program. It’s important for this group to focus the 
implications of these topics at the administrative level. 

Several questions can be asked about each of the quote: 

“I would like to suggest that developing performance accountability is 
not just technically challenging but also challenges our values” 

• What are the organization’s values about accountability? 

• What would it take to change the organization’s values? 

“ What is counted becomes what counts.” 

• Think about the Documentation Matrix. What is counted in your 
program? 

• Do you see ways that what is counted affects how you do your work? 

• Do people “teach to the test” and, if so, is that a problem? 

“Two kinds of capacity — to perform and to be accountable — are linked.” 

• How are these two capacities different? [Capacity to perform 
involves having the resources to meet your goals — time, trained 
staff, etc. Capacity to be accountable involves the collecting and 
reporting of data that measure performance, e.g., reliable testing, 
accurate reporting.] 

“In ABE, mutual accountability would engage members of the organiza- 
tion in creating a common vision, determining goals and customer 
expectations, and designing effective means of monitoring processes and 
results.” 

• What opportunities are currently in place for members of the orga- 
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nization to engage in visioning, determining goals and customer 
expectations? 

• What might get in the way of mutual accountability? 

It is important to document the outcome of this discussion for use in 
future planning. 




Step 3: Identifying Program Changes - 45 minutes 
In this step the group will list program changes that would lead to a more 
systematic approach to program improvements. The facilitator reminds 
participants that this is the last session before moving on to the planning 
phase and that much of the documentation from this phase will guide the 
planning for program changes. The facilitator and the participants can 
decide the most useful approach to this step. The group might choose one 
or the other option, a combination of both, or create another approach. 

Option 1. If the program has gone through the Baldrige National Quality 
Program, then ask participants to think about the seven Education Crite- 
ria for Performance Excellence and how each one might relate to, be 
enhanced by, or addressed through the five quality EFF Program Practices. 
The facilitators can document this discussion’s outcomes using an EFF 
and Baldrige matrix draw on a flipchart sheet. See the example of the 
matrix on the next page. Consider the session notes and documentation 
generated throughout this reflective process to complete the matrix. 

Option 2. If the program has not participated in or is not currently partic- 
ipating in the Baldrige National Quality Program, then have participants 
consider the session notes and documentation generated throughout this 
reflective process. 

Make a list of program changes to be made. Next, have participants priori- 
tize the list. 

The final part of the step is to think about a timeframe for the planning 
phase and to identify staff and stakeholders who will participate. Use a 
flipchart sheet to list potential participants. 
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Handout 2.21 



HANDOUT 2.21 

Education Criteria for Performance Excellence 
and EFF Program Practice 



Leadership 


B Purposeful and 
transparent 


Contextualized 

Approach 


Constructivist 

Approach 


Assessment 
Based on 
Cognitive 
Science 


Systematic 
Approach to 
Program 
Improvement 


Program leaders 
must understand 
the organizational 
purposes and 
goals. 


How can 
students build 
leadership 
skills through 
contextualized 
teaching and 
learning? 








Strategic 

Planning 




Our program 
must be 
thoughtful and 
strategic when 
planning to 
implement 
instruction that 
is contextualized. 








Customers 


How do our 

students 

understand 

the program’s 

purposes/goals 

and how do 

understand 

student’s 

purposes 

and goals? 










Information and 
Analysis 


What information 
do we need to 
collect and 
understand? 










Human 

Resources 












Process 

Management 












Results 













Step 4: Wrapping-Up -15 minutes 
Wrap up this session by restating the goals and objec- 
tives for this session and the goals and objectives for this 
phase of the project. Check in with participants to see if 
they feel the goals were met. Invite participants to share 
comments and reactions to this reflective phase of the 
process. 



Information to Save From 

Session Five 

• Important Points About 
Program’s Experience With 
Change 

• Notes from Discussion on 
Program Performance 
Accountability 

• List of Program Changes 
that Lead to a More 
Systematic Approach to 
Program Improvements 

•Timeframe for Planning 
Phase and List of Potential 
Participants 
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The EFF Handbook for Program Improvement is a tool based on Equipped 
for the Future to guide an organization in an examination of its adult edu- 
cation program quality and in a process of planning and implementing 
change. Phase 1 helps program administrators get ready for this process. 
Phase 2 is a set of activities based on the EFF Quality Model in which the 
program quality team assesses various aspects of their program. In Phase 
3 the team prioritizes the possible changes identified in Phase 2 and makes 
plans to implement one or more of these changes. In Phase 4 the program 
implements the planned changes, and the team documents and evaluates 
this implementation. 

The EFF Standards help structure the processes in this handbook. The 
facilitators of the program improvement process were asked to review the 
Cooperate with Others, Plan, and Reflect and Evaluate Standards as part of 
their preparation for this process. The Solve Problems and Make Decisions 
and Plan Standards frame the steps taken in Phases 2-4. Review the chart 
below. 



Standard 


Components of Standards 


Program 

Improvement 

Phase 


Solve Problems 
and Make Decisions 


Anticipate or identify problems 


Phase 2 


Use information from diverse sources to arrive at a clearer 
understanding of the problem and its root causes 


Phase 2 


Generate alternative solutions 


Phase 2 


Evaluate strengths and weaknesses of alternatives, including 
potential risks & benefits & short- and long-term consequences 


Phase 3 


Select alternative that is most appropriate to goal, context, and 
available resources 


Phase 3 


Establish criteria for evaluating effectiveness of solution or decision 


Phase 4 


Plan 


Set and prioritize goals 


Phase 2 


Develop an organized approach of activities and objectives 


Phase 3 


Actively carry out the plan 


Phase 4 


Monitor the plan’s progress while considering any need to 
adjust the plan 


Phase 4 


Evaluate its effectiveness in achieving the goals 


Phase 4 
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P lanning for change is a critical step in program improvement. 

In Phase 2 the program improvement team listed program 
goals. You identified problems or areas of practice that you 
wanted to improve. You used a variety of activities and sources 
of information to understand your program. You generated alternative 
solutions captured in “Changes to Make” lists. 



Handout 3.1 



In Phase 3 you will be 

addressing three compo- 

nents from the Solve Prob- 

lems and Plan Standards: 

• Evaluate strengths and weak- 
nesses of alternatives, including 
potential risks and benefits and 
short- and long-term conse- 
quences. 

• Select alternative that is most 
appropriate to goal, context, 
and available resources. 

• Develop an organized approach 
of activities and objectives. 

We will discuss each of these com- 
ponents of performance and how 
they support a thorough planning 
process. The program improve- 
ment team will prioritize your 



HANDOUT 3.1 

Steps in Phases 3 and 4 

From EFF Standards Solve Problems and Make Decisions and Plan 

PHASE 3 

• Evaluate strengths and weaknesses of alternatives. 

— revisit the initial reasons for engaging in quality improvement 

— revisit the organizational and program goals 

— prioritize the possible changes 

— consider the political, partnership, personnel, and financial realities of the organization 

• Select alternative that is most appropriate to goal, context, and available resources. 

— decide which program practice(s) to address and which changes to make 
— think about how the plan will be administered and the necessary funds 

• Develop an organized approach of activities and objectives. 

— determine what actions will be necessary to achieve these changes 

— think of all the people and organizations that need to be involved 

— outline the plan’s specific activities, responsibilities, a timeline for implementing 
the changes, and criteria for evaluation 

PHASE 4 

• Actively carry out the plan. 

— identify the responsibilities of team members 

— give someone the responsibility for making sure these supporting activities take place 
— be certain everyone knows their role 
— use your timeline to stay on track 

• Monitor the plan’s progress while considering any need to adjust the plan. 

— hold regular team meetings 

— document of the implementation of the plan 
— provide regular opportunities for review 

— record observations of unexpected occurrences 

• Establish criteria for evaluating effectiveness of solution or decision. 

— examine the documentation you have collected 

— identify the indicators of progress 

• Evaluate its effectiveness in achieving the goals. 

— collect data 

— analyze your data 
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goals and then evaluate the alternatives of your “Changes to Make” lists. 
You will select the alternative(s) you want to implement and develop a 
plan of objectives and activities to implement the chosen alternatives. 
The change(s) might involve one or all: staff, procedures, teaching and 
learning. 



Preparing for Phase 3 

To get ready for Phase 3, the team leader(s) will need to organize informa- 
tion and schedule meetings. All of the “Changes to Make Lists,” and other 
documentation generated from Phase 1 and 2 activities should be orga- 
nized, by EFF Program Practices, in a three-hole binder or folder. The 
number of meetings necessary for Phase 3 will be determined by the team; 
however, suggested are at least two, 2-hour planning meetings to be sched- 
uled with the program improvement team. 



Evaluate Strengths and Weaknesses of Alternatives 

The team will start by revisiting the initial reasons for engaging in quality 
improvement from Phase 1 (don’t be surprised if your initial reasons have 
shifted) and by reviewing all of the “Changes to Make Lists” from Phase 2. 
It will also be useful to revisit the organizational and program goals iden- 
tified in Session 1 of Phase 2. For example, one program goal might be to 
prepare students for good jobs, so that when they leave your program they 
are able to get living wage employment. Another goal might be to support 
your students to be more active as parents and family members in the 
education of their children. As a team, reaffirm these goals and determine 
which are priorities for you now. Use the Prioritization Activity in the 
Resource Appendix for help with this step. 

An important aspect of planning is analyzing the forces affecting the pro- 
gram’s ability to achieve its goals and objectives. In organizational devel- 
opment terms, this is often referred to as “force field” analysis. For the 
program improvement team this means taking time to consider the politi- 
cal, partnership, personnel, and financial realities of the organization. To 
analyze forces affecting your program, the team might brainstorm a list of 
forces working “for” and “against” your goals. The key question is “What 



AN EFF HANDBOOK FOR PROGRAM IMPROVEMENT 

62 



Phase 3: Planning for Program Improvement 



forces will affect our efforts to make planned changes, and how do we 
address them?” The points uncovered in this brainstorm will help guide 
the development of strategies to achieve goals. 



Select Changes That Are Most Appropriate to 
Goal, Context, and Available Resources 
In Phase 2 you listed possible “Changes to Make” for each EFF Program 
Practice. These changes are things that your team identified that could 
help you align your program components: 

• so that they will be more purposeful and transparent 

• so that teaching and learning are better aligned with learner and pro- 
gram goals and contexts, are designed to build expertise, and are 
assessed based on performance 

• so that you can document your work to be better accountable to 
internal and external stakeholders. 

Your list of “Changes to Make” may have included different approaches to 
recruitment, intake, orientation, curriculum and instruction, assessment, 
and program improvement. You will probably not want or be able to 
implement all these changes at one time. You will need to prioritize the 
possible changes and make deci- 
sions about which are most impor- 
tant in supporting your program 
goals and which are possible in 
your current program context. 

Whatever decisions are made 
should be reached by consensus to 
ensure that there is unity among 
team members and that change 
process can move smoothly. When 
making program or organizational 
changes, consensus building is a 
useful tool. Refer to the Steps in 
Consensus Decision-Making in the 
Resources Appendix. 



RESOURCES 

Steps in Consensus Decision-Making 

• Clarify the problem or question being addressed. 

• Determine the criteria for a good solution (e.g., cost, scale, scope, acceptance). 

• Ensure that everyone agrees on the criteria. 

• Brainstorm a range of alternative solutions. 

• Generate a thorough list of alternatives. 

Do not evaluate the alternatives as they are being generated — just list them. 

• Ensure that the ideas are written as they are visible to everyone. 

• Take a straw poll to check for the possibility of early agreement. 

• If there is only a little disagreement, determine the source of the disagreement. 

• Evaluate alternatives, according to the criteria. 

• Rule out alternatives that do not meet criteria or that on further 
reflection are rejected by the group. 

• Determine if any alternatives require further research. 

If so, come up with a plan for carrying out research. 

• Make a decision. 

• Test ways to combine or modify alternatives to meet the interests 
of as many people in the group as possible. 

• Select one or more alternatives. 
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As a team and with the program goals in mind, decide which program 
practice(s) to address and which changes to make. It will be important for 
the team to start small and select changes to make that are doable. For 
example, you might decide to work only on EFF Program Practice One — 
changes to make the organization’s approach to education more purpose- 
ful and transparent, so that students and potential students will know that 
you prepare program graduates to get living wage jobs. Alternatively, the 
team might decide to make only one change under each program practice. 



It will also be important to consider sustainability and funding as you 
make your choice. To ensure the long-term sustainability of your plan for 
improvement, the team will need to think about how the plan will be 
administered and where the 
necessary funds will come from 
in both the short-term and 
long-term. 

Develop an Organized 
Approach of Activities and 
Objectives 

Once you have decided which 
changes to make, determine 
what actions will be necessary to 
achieve these changes. To do this 
might require some additional 
research and background read- 
ing. For example you might 
want to change how you do 
assessment in your program so 
that your assessment is more 
authentic and learner centered, 
yet meets the requirements of 
the National Reporting System. 

Or you may decide to ask teach- 
ers to use the EFF Teaching/ 

Learning Cycle to more closely 
connect learner goals to instruc- 



The following links offer helpful information on 
program improvement. Team member tasks can 
include checking out these sites for useful 
information. 

www.nifl.gov/lincs/collections/eff/eff.html 

The National Institute for Literacy site offers 
specialized information on high quality literacy 
practices and materials for use in adult education 
and literacy programs. The EFF special collection 
includes downloadable versions of most EFF 
publications including masters of many EFF tools. 

www.quality.nist.gov/Education_Criteria.htm 

A link of the National Institute of Standards and 
Technology, this site offers the information on the 
Baldrige National Quality Program, which has 
developed education, health, and business criteria for 
organizational self-assessment and action planning. 

www.aelweb.vcu.edu/publications/Planning Guide/ 
planningguide2002.pdf 

Look for the professional Development Planning 
Guide, which outlines four EFF-specific professional 
development workshops. 

www.sabes.org/resources/adventures/vol13/ 

13rosenthal.htm. 

This volume of Adventures in Assessment features 
several articles that address issues of program 
accountability and assessment that is learner centered 
and authentic. 
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tion. Other organizations have gone down the path that your team is tak- 
ing and there are benefits to reading about and gathering information on 
the experiences of others. 

Before moving to the final step of crafting a plan, consider how stake- 
holders, staff and others who are critical to success are involved in the 
plan. As a team think of all the people and organizations that need to be 
involved. 

The final step in the planning phase is to outline the plan’s specific activi- 
ties, responsibilities, a timeline for implementing the changes, and criteria 
for evaluation. While there is much more about evaluation in Phase 4, it 
will, however, be important for the team to consider evaluation criteria 
and how the team will know that it has achieved its goals and objectives. 

The team can design its own planning grid or use and/or adapt the one 
below. 



Make sure that appropriate staff and partners have a copy of the plan with 
a timeline for accomplishing the activities and that everyone is on board 
with the changes to make. You are now ready to move on to Phase 4. 



Objectives 
(based on 
Changes to Make) 


Activities 


Specific 

Tasks 


Who is 
Responsible? 


Timeframe 


Criteria for 
Evaluation 
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Phase 4: Implementing, 
Documenting and Evaluating 
Program Changes 



P rogram improvement is an ongoing cycle of setting goals, 
reflecting, planning, implementing, evaluating, and reflecting. 
In Phases 1-3 of this handbook you reflected on your adult 
education program, set goals, and made plans. In Phase 4 you 
will implement your plan, document what you do and the results, and use 
your documentation to evaluate what you have done. 

In Phase 4 you will be addressing four components from the Solve Prob- 
lems and Plan Standards: 

• Actively carry out the plan. 

• Monitor the plans progress while considering any need to adjust the 
plan. 

• Establish criteria for evaluating effectiveness of solution or decision. 

• Evaluate its effectiveness in achieving the goals. 

We discuss these separately, but often they overlap in practice, as do 
implementation, documentation, and evaluation. 

As in Phase 3, this section provides suggestions to use as you address each 
component and suggestions for additional resources. 



Actively Carry Out the Plan 

In Phase 3 you developed a plan. In Phase 4 you will implement this plan, 
following the timeline that you have developed. You may be using a new 
intake procedure that helps students have clearer expectations and goals. 
You may have decided to encourage teachers to use the EFF role maps to 
plan with students learning activities that use authentic materials and 
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contexts. You may have planned to use the EFF Standards to better align 
student goals, instruction, and assessment. Or you may have decided on 
some other change based on your work in Phase 2. 

If you have not already done so as part of planning, identify the responsi- 
bilities of team members in this phase. Determine who will monitor 
implementation, who is responsible for documentation and evaluation. 

Whatever primary changes you have planned, you will also have planned a 
variety of supporting steps that you will take; for example, providing train- 
ing for intake personnel and teachers or restructuring your schedule to 
provide teachers more planning time. Implementation includes both the 
primary change and the supporting activities or changes. Give someone 
the responsibility for making sure these supporting activities take place. 

Be certain everyone knows their role in the implementation, whether 
major or minor. 

Use your timeline to stay on track. Make sure that everyone who might be 
affected is aware of the plan and knows the timeline. This includes stu- 
dents. 



Monitor the Plan’s Progress While Considering 
Any Need to Adjust the Plan 

Many EFF projects have found regular team meetings to be an important 
part of implementation. Having a regular time to check in with each other 
means that team members will be more likely to keep up to date on their 
part of the implementation. But more importantly, team meetings give 
people the opportunity to share their experiences, support each other, and 
to identify adjustments that may be needed. 

As you begin implementation, you may want to reconsider the make-up 
of your team. While most or all of the people involved in Phases 2 and 3 
could continue to be on the team in Phase 4, you may want to add other 
staff who have a major role in implementation. One teacher may have 
been a member of the original quality team, but all your teachers could be 
involved in implementation. Consider how extensive their role will be. 
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Will they be involved in documentation and evaluation as well as imple- 
mentation? They may need and want to be part of the quality team. Or, 
you may want to continue with the original quality team members and 
have the implementation team meet separately to reflect on the imple- 
mentation work. 

Monitoring also involves documentation of the implementation of the 
plan and any supporting activities. This documentation does not need to 
be elaborate, but does need to be consistent. In designing your documen- 
tation you may want to refer to the questions you considered in Session 4 
of Phase 2. This could help you avoid redundant documentation. These 
questions are: 

• What information will be documented? 

• By whom? 

• Using what method? 

• How often? 

• Who will use it? 

• For what purpose? 

In this component you are considering documentation of implementa- 
tion. You are documenting to determine if you are meeting your objec- 
tives in the plan. You want to be able to answer the question, “Did we do 
what we meant to do?” 

This documentation will be carried out primarily by those implementing 
the plan. The documentation method should be as time and effort effi- 
cient as possible. It may be as simple as keeping files of completed activi- 
ties or a log of the various activities undertaken by team members. It 
might be a checklist of the activities in the plan with space for dates and 
comments, or you may take photographs or record student projects. 

Whatever the method, there need to be regular opportunities for review - 
maybe at team meetings — to assure that people are consistent in using 
whatever approach you decide on. If there are serious problems with 
keeping up with documentation, this may be an indication that the meth- 
ods you have chosen are too cumbersome or do not match people’s work 
styles. You may want to adjust them. 
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In addition to monitoring the implementation of your plan, you want to 
be able to capture (document) What else happened? Journals are one way 
to record observations of unexpected occurrences, as well as a way to 
reflect personally on how a plan is working. You might keep a computer 
(or paper) file where you note ideas or new occurrences, even if you do 
not keep a regular journal. This might be expected of all team members, 
or could be one person’s responsibility. Another way to answer “what else 
happened?” is to include questions in regular team meetings that give peo- 
ple the opportunity to reflect on this. You will need to document this con- 
versation in some way, with notes, audio tape, etc. 



Establish Criteria for Evaluating 
Effectiveness of Solution or Decision 

In the planning process in Phase 3 you clarified your goals and objectives 
for implementing these particular changes. You determined the activities 
that would help you achieve these objectives and you identified your crite- 
ria for success — how you would know that your objectives have been met. 

Evaluation builds on this planning to address two questions: 

• What are the results of this implementation? Did we meet 
our objectives? 

• What was the impact? Did our results move us toward 
our goals? 

These questions are followed by “What next?” as your program takes part 
in a process of continuous improvement. 

The first step in determining the effectiveness of a plan or decision is to 
determine if the plan was carried out. You should be able to use your plan 
to examine the documentation you have collected as you monitor the 
implementation of the project. Have you met those criteria? Have you 
done what you planned to do? 

Ongoing evaluation while a project is being implemented is sometimes 
called formative evaluation. We are not suggesting a formal formative eval- 
uation, but as you monitor implementation you may want to refer to the 
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questions for formative evaluation in the Resource Appendix “Overview 
of Evaluation Prototypes.” 

Once you have determined what has happened, you will evaluate the 
results: Has the implemented change moved you toward your goals as an 
organization? To answer this question you need to identify the indicators 
of progress. What goal or goal did you expect that the change you imple- 
mented would impact? What indicators would tell you that you have 
made progress toward that goal? What was the situation regarding that 
goal before you began the implementation? What is it now? 

Here is an example of establishing criteria for effectiveness in regard to a 
goal: 

If an organization has as one of its goals to help prepare people 
for community college training programs and has determined 
that students having clear learning goals means they are more 
likely to be better prepared, the organization may have deter- 
mined as an objective to implement a student goal-setting 
process based on the Role Maps and Standards of EFF. After 
they implemented this EFF goal-setting process (which they 
determined had happened by ongoing monitoring and docu- 
mentation — formative evaluation), they wanted to evaluate the 
impact of this goal-setting. They identified as their criteria of 
success: 

• Students are able to describe their learning goals 

• Enrollment in community college training programs by their 
students increased 

• Completion of these programs on the part of their students 
increased. 

So the indicators that this program used to determine the success of 
implementing EFF-based goal-setting were students’ abilities to describe 
their learning goals and increase in community college enrollment and 
completion. This kind of evaluation that looks at the results of a program 
or implementation is often called a summative evaluation. In a s ummative 
evaluation you are looking at outcomes of a program or process and eval- 
uating the impact of your program or process. 
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The United Way of America website has examples of possible outcomes 
and indicators for various programs. 

http://national.unitedway.org.outcomes/resources/mpo/examples.cfm 



Evaluate Its Effectiveness in Achieving the Goals 

Once you have identified your indicators of progress or effectiveness, in 
order to evaluate effectiveness you will need to collect data to give you 
information on these criteria. Your evaluation will be most powerful if you 
have baseline data as well as data collected after a change is implemented. If 
your indicators include an increase or decrease in some factors, a baseline is 
essential. In the example above, the program might have interviewed stu- 
dents regarding their learning goals before the EFF goal-setting process was 
put in place and again after. They needed to have collected data on com- 
munity college enrollment and program completion by students from 
their program both before and after the EFF implementation. 

There are a wide variety of ways to collect and analyze data for evaluation. 
Because you will be evaluating one aspect of your program — the imple- 
mentation of a particular change — the process should not be too elabo- 
rate. By identifying the criteria you will use to determine effectiveness and 
collecting baseline data on these criteria, the evaluation should be reason- 
ably simple and efficient. Chapters Five and Six of Taking Stock: A Practical 
Guide to Evaluating Your Own Programs describe a variety of methods. 
Download from the Community Partnerships for Adult Teaming website 
at http://c-pal.net/build/assess/how.html. There are a variety of other eval- 
uation resources as this site, but Taking Stock is most accessible. 

After you have collected data on the outcomes of your implementation, 
you need to analyze your data to determine if the implementation was 
successful in moving you toward your program goals. Without a more 
elaborate evaluation than we are discussing here, it will probably not be 
possible to say with certainty that one change led to particular outcomes. 
But by examining your data and comparing them to your baseline data 
you should have a pretty good indication of whether you are moving in 
the direction you want to go. 
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After you complete your analysis, you will probably want to use your eval- 
uation as part of your program improvement efforts and, possibly, to 
report to external stakeholders. Create a simple report describing what 
you have done and the results. 
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Resources for Facilitators 



Resource List 

Stages of Team Development 
Prioritzing 

Steps in Consensus Decision Making 
Overview of Evaluation Prototypes 
Facilitation Guide 
Reference Materials 
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RESOURCES 

Resource List 



■ Baldrige National Quality Program 

National Institute of Standards and Technology 

100 Bureau Drive Stop 1020 

Gaithersburg, MD 20899-1020 

Phone: 301-975-2036 

Fax: 301-948.-3716 

Website: www.quality.nist.gov 

■ Center for Literacy Studies 

University of Tennessee 
600 Henley Street 
Knoxville, TN 37996-2135 
Phone: 865-974-6610 
Fax: 865-974-3857 
Website: www.cls.coe.utk.edu 



■ National Institute for Literacy 

1775 I Street, NW; Suite 730 
Washington, DC 20006-2401 
Phone: 202-233-2025 
Fax: 202-233-2050 
Website: www.nifl.gov 

■ National Institute for Literacy EFF Special 
Collection 

www.nifl.gov/linc/collections/eff/eff.html 

■ The National Urban League 

120 Wall Street 
New York, NY 1 0005 
Phone: 212-558-5300 
Fax: 212-344-5332 
Website: www.nul.org 



To learn more about the implementation of EFF in your program or state, contact Diane Gardner, EFF Training 
and Technical Assistance Center Coordinator at dgardner@utk.edu or 865-974-9949. 

For general information about obtaining EFF materials, training or technical assistance, contact Ginny Bleazey 
at the EFF Center for Training and Technical Assistance at bleazey@utk.edu or 865-974-8426. 

To learn more about the EFF assessment field research/development, contact Brenda Bell, Field Director of 
the EFF Assessment Consortium at bsbell@utk.edu or 865-974-6654. 

If you have technical questions about the development of the EFF Assessment Framework, contact Regie 
Stites, Technical Director of the EFF Assessment Consortium at regie.stites@sri.com or 650-859-3768. 

To learn more about the National Urban League’s collaboration with EFF, please contact Janet Zobel, Director, 
Program and Affliliate Development at jzobel@NUL.org or 212-558-5350. 

If you have questions about EFF as it relates to policy and legislation, please contact Sondra Stein, National 
Director of EFF at Sondra_Stein@nifl.gov or 202-233-2041 . 

You may also want to examine the following Baldrige Materials: 

• Getting Started with the Baldrige National Duality Program 

• E-Baldrige Self-Assessment and Action Planning, www.quality.nist.gov 

• Are We Making Progress? 

You can contact the Baldrige National Duality Program and the National EFF Center for these and other 
educational materials. The addresses are listed above. 
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Stages of Team Development 



STAGE 1 




Team Forming 



Team members are generally 
polite and guarded. Trust has 
not developed yet. 




STAGE 4 



Performing 



Team members work well 
together and are productive. 




STAGE 5 




STAGE 2 


Adjourning 


Storming 


Team members leave or new 
team members join and prepare 
to experience the stages 
again as a new team. 


Team members can display 
frustration with themselves, 
others, or the task at hand. 




1 


1 




STAGE 3 



Norming 



Team members get used 
to working with each other. 
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RESOURCES 

Prioritizing 



In organizational and group work, there comes a time when the group has to prioritize goals, events, 
strategies, etc. The following activity is a quick approach for prioritizing. 

The Dot Activity 

Goal: To engage group members in a process to identify priorities. 

Required 

Materials: Medium size self-adhesive sticky dots, markers, flipchart and pad 
Time: 10 to 20 minutes 

Step 1. 

Start with a list of topics, statements, issues, etc. This may be a brainstormed list or generated from 
another activity. Using bold makers, write the list on a flip chart sheet(s). When writing the list on the flip 
chart sheet, make sure enough space if left in the margins for sticky dots and/or other hand written notes. 

Step 2. 

Distribute self-adhesive sticky dots to group members. Decide how many dots each person should receive 
based on the number of items on the list and the number of people in the group. For example, if you 
have 5 items on the list and you want to identify the top three choices, each person might get three sticky 
dots. If you have a large group and you want to identify the top three, then each person might get one or 
two sticky dots. Each dot represents one vote. 

Step 3. 

Review the list with group members by giving some background and/or explanation on each item. 

Step 4. 

Now ask group members to place their sticky dots by the items they feel are most important or a priority. 

Step 5. 

Once all of the dots have been placed on the list, count the number of dots for each item, and make a new 
prioritized list, with the items getting the most dots at the top. 

A word of caution - sometimes, the items that get the most dots are not necessarily the best choices. This 
activity is useful to a group as a first step in a broader conversation about a particular decision, solution, 
plan, etc. Once the larger list is prioritized, then, the group might have a follow-up conversation about 
the strengths, weaknesses, support for and against any one item. 
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Steps in Consensus Decision-Making 



• Clarify the problem or question being addressed. 

• Determine the criteria for a good solution (e.g., cost, scale, scope, acceptance). 

• Ensure that everyone agrees on the criteria. 

• Brainstorm a range of alternative solutions. 

• Generate a thorough list of alternatives. 

Do not evaluate the alternatives as they are being generated — just list them. 

• Ensure that the ideas are written as they are visible to everyone. 

• Take a straw poll to check for the possibility of early agreement. 

• If there is only a little disagreement, determine the source of the disagreement. 

• Evaluate alternatives, according to the criteria. 

• Rule out alternatives that do not meet criteria or that on further 
reflection are rejected by the group. 

• Determine if any alternatives require further research. 

If so, come up with a plan for carrying out research. 

• Make a decision. 

• Test ways to combine or modify alternatives to meet the interests 
of as many people in the group as possible. 

• Select one or more alternatives. 
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Overview of Evaluation Prototypes 



Planning Evaluation 

A Planning Evaluation assesses the understanding of project goals, objectives 

strategies and timelines. 

It addresses the following types of questions: 

• Why was the project developed? What is the problem or need it is attempting to 
address? 

• Who are the stakeholders? Who are the people involved in the project? Who are the 
people interested in the project who may not be involved? 

• What do the stakeholders want to know? What questions are most important to 
which stakeholders? What questions are secondary in importance? Where do 
concerns coincide? Where are they in conflict? 

• Who are the participants to be served? 

• What are the activities and strategies that will involve the participants? What is the 
intervention? How will participants benefit? What are the expected outcomes? 

• Where will the program be located (educational level, geographical area)? 

• How many months of the school year or calendar year will the program operate? 
When will the program begin and end? 

• How much does it cost? What is the budget for the program? What human, material 
and institutional resources are needed? How much is needed for evaluation? for 
dissemination? 

• What are the measurable outcomes? What is the expected impact for the project in 
the short run? the longer run? 

• What arrangements have been made for data collection? What are the understand- 
ings regarding record keeping, responding to surveys and participation in testing? 



Formative Evaluation 

A Formative Evaluation assesses ongoing project activities. It consists of two types: 
Implementation Evaluation and Progress Evaluation. 

Implementation Evaluation 

An Implementation Evaluation assesses whether the project is being conducted as 
planned. It addresses the following types of questions: 
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• Were the appropriate participants selected and involved in the planned activities? 

• Do the activities and strategies match those described in the plan? If not, are the 
changes in activities justified and described? 

• Were the appropriate staff members hired, and trained, and are they working in 
accordance with the proposed plan? Were the appropriate materials and equipment 
obtained? 

• Were activities conducted according to the proposed timeline? by appropriate per- 
sonnel? 

• Was a management plan developed and followed? 

Progress Evaluation 

A Progress Evaluation assesses the progress made by the participates in meeting the 

project goals. It addresses the following types of questions: 

• Are the participants moving toward the anticipated goals of the project? 

• Which of the activities and strategies are aiding the participants to move toward the 
goals. 



Summative Evaluation 

A Summative Evaluation assesses project success — the extent to which the completed 
project has met its goals. It addresses the following types of questions: 

• Was the project successful? 

• Did the project meet the overall goal(s)? 

• Did the participants benefit from the project? 

• What components were the most effective? 

• Were the results worth the project’s cost? 

• Is this project replicable and transportable? 

From the EHR/NSF Evaluation Handbook 

http://www.ehr.nsf.gov/rec/programs/evaluation/handbook 
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Facilitation Guide 



Key Facilitation Skills 

Reflecting - feeding back the content and feeling of the message. 

“Let me see if I’m hearing you correctly. . . ” 

Clarifying - restating an idea or thought to make it more clear. 

“What I believe you are saying is. . . ” 

Summarizing - stating concisely the main thoughts. 

“It sounds to me as if we have been talking about a few major themes. . . ” 

Shifting focus - moving from one speaker or topic to another. 

“Thank you, John. Do you have anything to add, Jane?” 

“We’ve been focusing on views 1 and 2. Does anyone have strong feeling about the other 
views?” 

Using silence - allowing time and space for reflection by pausing between comments. 

Using non-verbal and verbal signals - combining body language and speech to 
communicate — for example, using eye contact to encourage or discourage behaviors 
in the group. Be aware of cultural differences. 

Neutrality is important here, so that we don’t encourage some people more 
than others. 



From Study Circle Resource Center, RO. Box 203, Pomfret, CT 06258 
Phone:860-928-2616 Fax:860-928-3713 Email: scrc@neca.com 
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Facilitation Guide 

Tips for Effective Discussion Facilitation 

Be prepared. 

The facilitator does not need to be an expert on the topic being discussed, but should 
be prepared for the discussion. This means understanding the subject, being familiar 
with the discussion materials, thinking ahead of time about the directions in which 
the discussion might go, and preparing questions to help further the discussion. 

Set a relaxed and open tone. 

• Welcome everyone and create a friendly and relaxed atmosphere. 

• Well-placed humor is always welcome, and helps to build the group’s connections. 

Establish clear ground rules. 

At the beginning of the study circle, help the group establish its own ground rules by 
asking the participants to suggest ways for the group to behave. Here are some ground 
rules that are tried and true: 

• Everyone gets a fair hearing. 

• Seek first to understand, then to be understood. 

• One person speaks at a time. 

• Share “air time.” 

• Conflict is not personalized. Don’t label, stereotype, or call people names. 

• Speak for yourself, not for others. 

• What is said in this group stays here, unless everyone agrees to change that. 

Monitor and assist the group process. 

• Keep track of how the group members are participating — who has spoken, who 
hasn’t spoken, and whose points haven’t been heard. 

• Consider splitting up into smaller groups to examine a variety of viewpoints or to 
give people a chance to talk more easily about their personal connection to the issue. 

• When deciding whether to intervene, lean toward non-intervention. 

• Don’t talk after each comment or answer every question; allow participants to 
respond directly to each other. 

• Allow time for pauses and silence. People need time to reflect and respond. 

• Don’t let anyone dominate; try to involve everyone. 

• Remember: a study circle is not a debate, but a group dialogue. If participants forget 
this, don’t hesitate to ask the group to help re-establish the ground rules. 
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Help the group grapple with the content. 

• Make sure the group considers a wide range of views. Ask the group to think about 
the advantages and disadvantages of different ways of looking at an issue or solving 
a problem. 

• Ask participants to think about the concerns and values that underlie their beliefs 
and the opinions of others. 

• Help the discussion along by clarifying, paraphrasing, and summarizing 
the discussion. 

• Help participants to identify “common ground,” but don’t try to force con 

Use probing comments and open-ended questions which don’t lead to yes or no 
answers. This will result in a more productive discussion. Some useful questions include: 

• What seems to be the key point here? 

• What is the crux of your disagreement? 

• What would you say to support (or challenge) that point? 

• Please give an example or describe a personal experience to illustrate that point. 

• Could you help us understand the reasons behind your opinion? 

• What experiences or beliefs might lead a person to support that point of view? 

• What do you think people who hold that opinion care deeply about? 

• What would be a strong case against what you just said? 

• What do you find most persuasive about that point of view? 

• What is it about that position that you just cannot live with? 

• What have we missed that we need to talk about? 

• What information supports that point of view? 

Reserve adequate time for closing the discussion. 

• Ask the group for last comments and thoughts about the subject. 

• Thank everyone for their contributions. 

• Provide some time for the group to evaluate the study circle process. 
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RESOURCES 

Facilitation Guide 

Suggestions for Dealing With Typical Challenges 

Most study circles go smoothly because participants are there voluntarily and have a 
stake in the program. But there are challenges in any group process. What follows are 
some of the most common difficulties that study circle leaders encounter, along with 
some possible ways to deal with those difficulties. 



PROBLEM: 

Certain participants 
don’t day anything, 
seem shy. 


POSSIBLE RESPONSES: 

Try to draw out quiet participants, but don’t put them on the 
spot. Make eye contact — it reminds them that you’d like to hear 
from them. Look for nonverbal cues that indicate participants 
are ready to speak. Frequently, people will feel more comfort- 
able in later sessions of a study circle program and will begin 
to participate. When someone comes forward with a brief 
comment after staying in the background for most of the study 
circle, you can encourage him or her by conveying genuine 
interest and asking for more information. And it’s always helpful 
to talk with people informally before and after the session. 


PROBLEM: 

An aggressive or 
talkative person 
dominates the 
discussion. 


POSSIBLE RESPONSES: 

As the facilitator, it is your responsibility to handle domineering 
participants. Once it becomes clear what this person is doing, 
you must intervene and set limits. Start by limiting your eye 
contact with the speaker. Remind the group that everyone is 
invited to participate; “Let’s hear from some folks who haven’t 
had a chance to speak yet.” If necessary, you can speak to the 
person by name. “Charlie, we’ve heard from you; now let’s hear 
what Barbara has to say.” Be careful to manage your comments 
and tone of voice — you are trying to make a point without 
offending the speaker. 


PROBLEM: 

Lack of focus, not 
moving forward, 
participants wander 
off the topic. 


POSSIBLE RESPONSES: 

Responding to this takes judgement and intuition. It is the 
facilitator’s role to help move the discussion along. But it is 
not always clear which way it is going. Keep an eye on the 
participants to see how engaged they are, and if you are in 
doubt, check it out with the group. “We’re a little off topic right 
now. Would you like to stay with this, or move on to the next 
question?” If a participant goes into a lengthy digression, you 
may have to say, “We are wandering off the subject, and I’d 
like to give other a chance to speak.” 
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PROBLEM: 

Someone puts forth 
information which you 
know to be false. Or 
participants get hung 
us in a dispute about 
facts but no one 
present knows the 
answer. 


POSSIBLE RESPONSES: 

Ask, “Has anyone heard of conflicting information?” 

If no one offers a correction, offer one yourself. And if 
no one knows the facts, and the point is not essential, 
put it aside and move on. If the point is central to the 
discussion, encourage members to look up the 
information before the next meeting. Remind the 
group that experts often disagree. 


PROBLEM: 

Lack of interest, no 
excitement, no one 
wants to talk, only a 
few participating. 


POSSIBLE RESPONSES: 

This rarely happens in study circles, but it may occur if 
the facilitator talks too much or does not give participants 
enough time to respond to questions. People need time 
to think, reflect, and get ready to speak up. It may help to 
pose a question and go around the circle until everyone 
has a chance to respond. Occasionally, you might have a 
lack of excitement in the discussion because the group 
seems to be in agreement and isn’t coming to grips 
with the tensions inherent in the issue. In this case, 
the leader’s job is to try to bring other views into the 
discussion, especially if no one in the group holds them. 
“Do you know people who hold other views? What 
would they say about our conversation?” 


PROBLEM: 

Tension or open 
conflict in the group. 
Perhaps two 
participants lock 
horns and argue. 

Or, one participant 
gets angry and 
confronts another. 


POSSIBLE RESPONSES: 

If there is tension, address it directly. Remind participants 
that disagreement and conflict of ideas is what a study 
circle is all about. Explain that, for conflict of ideas to 
be productive, it must be focused on the issue: it is 
acceptable to challenge someone’s ideas, but personal 
attacks are not acceptable. You must interrupt personal 
attacks, name-calling, or put-downs as soon as they 
occur. You will be better able to do so if you have 
established ground rules that disallow such behaviors 
and that encourage tolerance for all views. Don’t hesitate 
to appeal to the group for help; if group members bought 
into the ground rules, they will support you. As a last 
resort, consider taking a break to change the energy 
in the room. You can take the opportunity to talk 
one-on-one with the participants in question. 
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Appendix B: 
Session Handouts 



2.1 Introduction to EFF 

2.2 The Equipped for the Future Content Framework 

2.3 EXAMPLE: Goals and Broad Areas of Responsibilities 

2.4 Goals Grid 

2.5 A Purposeful and Transparent Approach to Education: Background and 
Case in Point 2 

2.6 EFF Research to Practice Note 1 

2.7 A Contextualized Approach to Curriculum and Instruction: Background and 
Case in Point 3 

2.8 EFF Research to Practice Note 2 

2.9 EFF Research to Practice Note 3 

2.10 Constructivist Learning 

2.1 1 EFF Teaching and Learning Cycle 

2.12 EFF Teaching and Learning Cycle Grid 

2.1 3 Assessment in the EFF Classroom 

2.14 Assessment, Accountability and Program Improvement: Background and 
Case in Point 4 

2.1 5 Questions on Assessment 

2.1 6 Current Approach to Assessment 

2.1 7 Traditional Versus EFF Approaches to Assessment 

2.1 8 Article by Juliet Merrifield 

2.19 Accountability, Change and Quality: Background and Case in Point 5 

2.20 Questions About Change 

2.21 Education Criteria for Performance Excellence and EFF Program Practice 
3.1 Steps in Phases 3 and 4 
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HANDOUT 2.1 

Introduction to EFF 

In the initial session of this program improvement process, team members are introduced to EFF as a 
standards-based framework for guiding the delivery of quality adult basic skills education. A quality pro- 
gram is one in which the organization’s purposes and goals are clear, so that students and community 
members can understand how a program or organization contributes to the students’ goals and to the 
broader community’s goals. 

EFF was initiated in 1993 in response to National Educational Goal 6: 

Every adult American will be literate and possess the knowledge and skills necessary to 
compete in a global economy and exercise the rights and responsibility of citizenship. 

By clearly defining what adults need to know to achieve this goal, EFF makes it possible for the adult liter- 
acy system to focus on measurable standards so that we can determine progress toward achieving Goal 6. 

In research with adult students, EFF has identified Four Purposes for Learning 

• to gain access to information and resources to orient themselves in the world, 

• to give voice to ideas and opinions with the confidence that they will be heard, 

• to take independent action to solve problems and make decisions, and 

• to keep learning to build a bridge to the future in a rapidly changing world. 

EFF helps organizations enable adults to achieve these four purposes in their roles as citizens, workers, 
and family members by providing 

• a clear set of 16 Content Standards that describe what adults need to know and be able to do 

• a framework and tools that teachers and programs can use to link curriculum, instruction and assess- 
ment 

• an assessment framework that can be used to support program improvement and accountability. 

The book, Equipped for the Future Content Standards: What Adults Need to Know and Be Able to Do in the 
21st Century (Stein, 2000) summarizes EFF history and goals, defines the key elements of EFF, defines the 
EFF Content Standards, and gives examples of how the EFF Framework has been used. 

The five EFF Program Practices reflect the theoretical foundations of EFF. These practices are explained 
in Results That Matter: An Approach to Program Quality Using Equipped for the Future (Bingman and 
Stein, 2001). See the Contacts list in the Appendix for ordering information. 
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HANDOUT 2.2 



The Equipped for the Future Content Framework 



Four Purposes 


Role Maps 


Common Activities 


Content Standards I 


To gain 


Citizens and 


• Gather, analyze and 


Communication Skills 


ACCESS 


Community Members 


use information 


• Read With Understanding 

• Convey Ideas in Writing 






• Manage Resources 


• Speak So Others Can 








Understand 






• Work within the big 


• Listen Actively 






picture 




To give 






Decision-Making Skills 


VOICE 




• Work together 


• Solve Problems and Make 




Parents and Family 




Decisions 




Members 


• Provide leadership 


• Plan 

• Use Math to Solve 






• Guide and support 


Problems and 






others 


Communicate 


To take 




• Seek guidance and 


Interpersonal Skills 


INDEPENDENT 




support from others 


• Cooperate With Others 


ACTION 


Workers 


• Develop and express 


• Guide Others 
•Advocate and Influence 






sense of self 


• Resolve Conflict and 








Negotiate 






• Respect others and 








value diversity 


Lifelong Learning Skills 

• Take Responsibility for 


To build a 




• Exercise rights and 


Learning 


BRIDGE TO 




responsibilities 


• Learn Through Research 


THE FUTURE 




• Create and pursue 


• Reflect and eEvaluate 

• Use Information and 






vision and goals 


Communication 

Technology 






• Use technology 








and other tools to 
accomplish goals 








• Keep pace with 








change 
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HANDOUT 2.3 

EXAMPLE: Goals and Broad Areas of Responsibility 

My personal goal is to create a small, home-based arts business that I can work 
and operate in my spare time. When I looked at the Worker Role Map, I located 
my goal under the Broad Area of Responsibility — “Plan and Direct Personal 
and Professional Growth.” When I looked closer at the Plan and Direct Personal 
and Professional Growth” section of the Role Map, I found the following Key 
Activities: 

• Balance and support work, career, and personal needs. 

• Pursue work activities that provide personal satisfaction and meaning. 

• Plan, renew, and pursue personal and career goals. 

• Learn new skills. 

So, these are four key activities that I need to “perform” so that I’m prepared as a 
small, home-based business “worker.” 



AN EFF HANDBOOK FOR PROGRAM IMPROVEMENT 

91 



HANDOUT 2.4 

Goals Grid 



GOALS 


Students 


Staff 


Program 
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HANDOUT 2.5 

A Purposeful and Transparent Approach to Education: 
Background and Case in Point 2 



Background: 

A quality program is one in which the organization’s purposes and goals are clear. It is important for students 
and community members to understand how a program or organization relates and contributes to the student’s 
goals and to the broader community’s goals. Intake and orientation processes that are purposeful and transpar- 
ent contribute to program quality by helping students to clarify what they need to learn in order to meet their 
goals. 

Educational theory and research says that learning is most effective when it is purposeful, when it connects to 
the attitudes, knowledge, skills, and behaviors that students need and want to fulfill their goals, and when it 
helps them become competent in their many adult roles. The purposes of the teaching and learning activities 
should be clear or transparent to the learner, teacher, and program. For additional background information on a 
purposeful and transparent approach to education, see EFF Research to Practice Note 1 in the Appendix. 
Administrators of an adult education program using EFF and wanting to improve program quality should consid- 
er the question: 

How will this agency develop a purposeful and transparent approach to education? 

One way in which a program is purposeful and transparent is when its program goals are clear and understand- 
able. In this session, the team members review promotional and other organizational materials and reflect on 
current intake and orientation processes to identify changes that would make these processes more purposeful, 
transparent, and useful. 



Case in Point: 

An eleven-year-old community based organization, The Pierce CommunityBuild Adult Education Cen- 
ter serves mostly young (18 to 25), low-income adults. The program offers students an opportunity to 
obtain a GED, learn a building trade, and gain valuable life skills such as budgeting and civics. Most of the 
students come to the program to get their GED, and they become frustrated when instruction is not tied 
directly to GED preparation. 

The program has decided that it needs to present a clearer picture to potential students about the pro- 
gram’s purposes and that it has to get a clearer picture from students about their learning purposes and 
their life goals. In fact, all the processes connected to the teaching and learning (intake, placement testing, 
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goal setting, selecting instructional content, and on-going assessment) need to be clear and understand- 
able — they need to be transparent. Both teachers and students need to understand the program’s process- 
es, instructional approach, and expected outcomes. 

Ms. Franklin, the program teacher, says, “When something is transparent, you can see through it.” She 
recently heard about the EFF Content Standards Framework at a conference. She was reminded that 
adults need to exercise their rights and responsibilities as family members, workers, and community 
members, and that EFF had identified four specific purposes for which adults say they seek education: 

1. To access information 

2. To voice their own ideas and opinions 

3. To take independent action to solve problems, and 

4. To create a bridge to the future. 

Ms. Franklin wondered how these purposes compared with what students say when they come to the 
CommunityBuild program. So, Ms. Franklin had her class do a writing assignment in which students 
identified their individual purposes and compared them to the four EFF purposes. Many of the students 
wanted to be able to “take independent action to solve problems.” 

The insights revealed in the writing activity were useful to both Ms. Franklin and to Mr. Johns, the educa- 
tional coordinator. They were able to identify a range of student purposes and goals that they could then 
review in light of the program’s purposes and goals. 
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HANDOUT 2.6 



EFF RESEARCH TO PRACTICE NOTE 




EFF Research Principle: 

A Purposeful and Transparent Approach 
to Teaching and Teaming 



By Marilyn K. Gillespie 




What Do We Mean by a Purposeful and Transparent Approach? 



T he first key research principle underlying the Equipped for the Future 
system reform initiative emphasizes its purposeful approach to teach- 
ing and learning. Purposeful learning is integral to every step of the 
teaching and learning process. The first step in EFF-based instruction 
involves asking learners to examine their broad purposes for learning 
in relationship to their roles as workers, as parents and family members, and as cit- 
izens and community members. The EFF Role Maps and the Common Activities 
that encompass all three roles provide a common language to talk about these 
broader “big-picture” purposes. From this base, adults can identify more specific 
goals that will allow them to achieve those purposes. The EFF Content Standards 
provide a guide to the knowledge, skills, and learning strategies that learners will 
need to reach the goals. Teachers and learners then decide together on specific 
learning activities that will enable learners to strengthen their knowledge and skills 
in the EFF Standard or Standards that are most critical to achievement of their 
goals. This intentional and purpose-driven approach to planning creates the con- 
ditions for teachers to make explicit both what will be learned and what good per- 
formance will look like. In this way, the process and goals of learning are 
transparent to everyone involved. 

Since the tools that make up the EFF Framework were developed through a broad 
national consensus-building process involving hundreds of adult learners, they 
connect individual learner goals to the broader, more fundamental purposes of 
the larger community. These include our National Goal for Fiteracy and Fifelong 
beaming, that every adult American will be literate and will possess the knowledge 
and skills necessary to compete in a global economy and exercise the rights and 
responsibilities of citizenship. 

This Research to Practice Note summarizes the growing body of research that sup- 
ports a purposeful and transparent approach to learning, including findings that 
demonstrate that: 

• Learning itself is a purposeful, goal-directed activity. An ongoing goal- 
setting process is integral to effective learning. 

• Purposeful and transparent learning builds on learners’ prior knowledge and 
experiences to construct new knowledge. 

• Purposeful and transparent learning also means that learners monitor and 
assess their own progress. Metacognitive strategies help them to be mindful of 
what is being learned and what good performance looks like. 



The EFF publication Results 
That Matter: An EFF Approach 
to Quality presents five key prin- 
ciples that reflect the theoretical 
foundations of EFF. Program 
practices that support these 
principles provide guideposts 
by which programs, teachers, 
students, and their communities 
can assess their implementation 
of the EFF Framework. They 
help practitioners to better 
answer the questions “What 
does it mean to practice EFF?” 
and "What does EFF implemen- 
tation look like in action?” 

These Research to Practice 
Notes will help you to: 

• identify the research basis for 
the principles; 

• learn key concepts and terms 
associated with the principles; 

• see examples of how other 
programs have implemented 
the program practices; 

• reflect on how you and your 
program can implement the 
program practices. 



National Institute for Literacy 

1775 I Street NW, Suite 730 
Washington, DC 20006 
TEL 202.233.2025 
FAX 202.233.2050 
WEB www.nifl.gov 
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EFF RESEARCH TO PRACTICE NOTE 1 



How Cognitive Science 
Informs EFF 

Researchers have recently made 
great progress in understanding 
how people think and learn. In the 
last few years, there has been an 
extraordinary growth in scientific 
work on the mind and brain. We 
now have ways to study not only 
the products of thinking and 
learning but also the processes 
by which people acquire new 
information, such as the neural 
processes that occur during 
thought and learning and the 
process through which people 
develop competence and exper- 
tise. The multidisciplinary group of 
researchers who have conducted 
this work have coined a term for 
the study of thinking and learning: 
cognitive science. Cognitive 
science research represents one 
of the key conceptual underpin- 
nings of the EFF Framework for 
teaching, learning, and assess- 
ment and provides the research 
basis for the constructivist theory 
of learning. For teachers who 
would like to read more about 
cognitive science research, the 
National Academy of Sciences 
has produced three publications 
that synthesize this work and its 
implications for education: How 
People Learn: Brain, Mind, Experi- 
ence, and School, by Bransford, 
Brown, and Cocking (1 999); How 
People Learn: Bridging Research 
and Practice, by Donovan, Brans- 
ford, and Pellegrino (1999); and 
Knowing What Students Know: 
The Science and Design of 
Educational Assessment, by 
Pellegrino, Chudowsky, and 
Glaser (2001). 



What Research Says about Purposeful and Transparent Learning 



Learning is a purposeful, goal-directed activity. 

Within the field of cognitive science, meaning is understood as something we 
impose on the world, rather than something that exists independently of our- 
selves. Beginning with Piaget, numerous researchers have demonstrated that 
human beings are, by nature, active problem solvers who seek out learning in 
order to make meaning of the world around them (Piaget, 1970; Bruner, 1986; 
Fosnot, 1992; Wenger, 1998). We undertake learning activities “not merely as 
ends in themselves but as means for achieving larger objectives and goals that 
have meaning in the community” (Scribner, 1987). Learning is a process of 
constructing new knowledge on the basis of our current knowledge to meet 
our socially determined purposes for learning (Glaser, 1992; Duffy & 
Jonassen, 1992). 

Goal setting and persistence. In light of this research, more attention is now 
being given to involving students in setting their own goals for learning. Within 
adult education, the role of goal setting has been underscored by recent research 
on what helps adult literacy and English for Speakers of Other Languages 
(ESOL) learners to persist in adult education programs. As they followed adult 
learners over time, Comings, Parrella, and Soricone (2000) found that adults 
who were able to identify more clearly their purposes for learning, such as “help 
my children” or “get a better job,” were much more likely to persist than those 
who either mentioned no specific purpose or simply said they were doing it for 
themselves. According to these researchers, learners who establish concrete goals 
and are given the opportunity to see that they are making measurable progress 
toward them are more able to persist in their efforts (and stay in programs) long 
enough to reach them. 

Purposeful learning in the EFF Content Framework. Programs using EFF 
build on these research findings by helping learners to identify their purposes 
and goals for learning at multiple stages in the teaching and learning cycle. Like 
most programs, they ask learners to name their goals during initial program 
intake, but this is only the beginning of a more extensive goal-setting process. 
Students are introduced to the EFF Framework, including the four Purposes for 
Learning, the Role Maps, and the Common Activities. These provide a com- 
mon language to help learners create a detailed “big picture” of their underlying 
purposes for learning. For example, students who name “getting my GED” are 
encouraged to look beyond passing the test to examine the goals it will help 
them to reach. If their goals are within the worker role, they may explore the 
need for postsecondary job training and then use the EFF Standards Wheel to 
determine that they need to develop skills in the Content Standards Plan and 
Learn Through Research to get into and succeed in community college. In a 
group learning situation, the group members work to reach a consensus on 
shared priorities. Together with their teacher, they then plan learning activities 
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EFF RESEARCH TO PRACTICE NOTE 1 



For reflection. . . 

• What kind of goal setting takes place 
within your program ? 

• How are your students involved in 
setting the goals and activities 
they will work on? 



that will help them work on those goals. At 
this stage, learners also become involved in 
the identification of what good perfor- 
mance related to their goals will look like. 
In this manner, activities become learner 
driven, transparent, and purposeful. 



Purposeful learning builds on learners’ prior knowledge. 

Cognitive research has shown that learning is not simply a process of “knowledge 
acquisition” but an active process of “knowledge construction” in which learners 
use their prior knowledge and experience to shape meaning and construct new 
knowledge (Lambert & Walker, 1995). (See Research to Practice Note 2 for more 
details on constructivist teaching and learning.) Teachers need to activate the 
prior knowledge of learners by helping them to articulate what they already know 
about a given topic and build on these ideas in ways that help students achieve a 
more expert understanding. For example, research with children has shown that 
when many young students are taught that the earth is round, they often do not 
fully grasp this new information. Many hold onto a mental model of the earth as 
flat by imagining a round earth to be shaped like a pancake ( Vosniadou & Brewer, 
1989). Only when teachers directly address learners’ prior knowledge of the earth 
as being flat are they able to help students develop a more complete understanding 
of the shape of the earth. If teachers do not involve learners in naming and analyz- 
ing those prior conceptions, research shows that students may fail to grasp the new 
concepts or may learn them for a test but revert to their preconceptions outside 
the classroom (Donovan, Bransford, & Pellegrino, 1999). 



Goal setting and self-assessment. Within the EFF teaching and learning cycle, once 
learners’ goals have been established, they begin a process of self-assessment in rela- 
tion to the EFF Content Standard they have decided to work on. They may first use 
For reflection brainstorming or other techniques to name and validate, 

• How could you use the EFF 
Framework to begin to work with 



students with limited literacy skills? 

In what ways do you find out about the 
prior knowledge of learners you teach? 
How might you use the EFF Framework 
to help to draw on and address their 



as individuals and as a group, what they already 
know. This process helps them to begin to 
examine and revise their existing mental 
models of the subject matter. For exam- 
ple, learners who think writing is mostly 
about “spelling all the words right” may 
learn that spelling is only one part of a 



prior knowledge? 



much larger writing “process.” 



Purposeful and transparent learning means 

that learners monitor and assess their own progress. 

A purposeful and transparent approach to teaching and learning requires that 
learners have a clear understanding of the purposes for each learning activity and 
monitor their own progress. For example, the Standard Listen Actively includes 
monitoring comprehension and integrating information from listening with prior 



In Other People’s Words: The 
Cycle of Low Literacy (1 995), 
Purcell-Gates chronicles the 
story of what can happen when 
literacy learning is divorced from 
broader purposes and everyday 
roles. Jenny, a white urban 
Appalachian mother, came to 
Purcell-Gates for help with 
literacy for herself and her son. 
At 31 , she and her husband 
had created a full life for them- 
selves, but one in which literacy 
played very little part. When 
Purcell-Gates met her, Jenny 
had been attending adult 
education classes off and on 
for four years. She showed 
Purcell-Gates her books, which 
contained short reading 
passages, comprehension 
questions, and fill-in-the-blank 
exercises. Although she was 
able to read workbooks written 
at the fourth-grade reading 
level, she had transferred none 
of this knowledge to her every- 
day life. She had never written 
anything on her own, for her 
own purposes, besides her 
name, a few notations on the 
calendar, and her address. 
When Purcell-Gates suggested 
to Jenny that she write in a 
journal and read her own writ- 
ing, “She looked at me with an 
expression of stunned aware- 
ness. ‘Why, I ain’t never read 
my own words before!’ she 
exclaimed softly... ‘That’s all I 
ever really did was copy stuff, 
you know, from a book.’” 
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I found I had to be much more keenly 
aware of where my learners began and then 
where they are now. I felt much more in 
touch with what my learners needed ( time 
to reflect or for the peer leadership that 
occurred) and felt more like it was a 
respected classroom. I’ve always been 
unsure of what student- centered is, and 
then this summer’s group really showed me 
how the balance needs to happen between 
teacher and learner. This process also has 
made me understand how important it is 
to know where the learner begins through 
observation or through learner assessment. 

I used their strengths to plan and 
incorporate discussions where before I 
wouldn’t have been as aware of the process 
the learners were going through, and I 
worried more about how I was doing 
rather than how they were progressing. 

— Jennifer Ladd, Atkinson, Maine 

This task was successful because the 
students inspired it. They were invested in 
this activity [writing to request funds for 
community service]. They were hopeful we 
would see results. They could see the value 
in learning to write a business letter, and 
most of them felt they would be able to 
write letters on their own in the future. . . 
The students would pepper me with 
questions about our project. They wanted 
to know if I’d sent the letters or if I’d heard 
any response. They were talking about it 
every day. 

— Joanna Elizondo, Seattle, Wasliington 



For reflection. . . 

• Think about your own metacognitive processes. What 
kinds of strategies do you use to monitor and assess your 
own learning as a teacher? 

• Look at the Components of Performance for several EFF 
Standards. How is metacognitive awareness built into 
the language of the components? 



knowledge to address the listening purpose. One way teachers help 
learners to do this is by improving their awareness of metacognitive 
processes of learning. 

The importance of metacognitive awareness. Metacognition refers 
to our capacity to be aware of our own thinking processes and to 
monitor and control our thinking relative to the cognitive tasks we 
are performing (Greeno, Resnick, & Collins, 1997). For example, you 
are using your metacognitive skills when you monitor your under- 
standing while you are reading, when you go back and reread pas- 
sages you don’t understand, and when you decide when and under 
what conditions to consult a dictionary. Cognitive research has 
demonstrated that most experts have strong metacognitive skills in 
relation to their field of expertise. They make “mental notes” when 
they need more information. They observe whether what they are 
learning is consistent with what they already know, and they monitor 
what they are learning to see whether it meets their purposes (Brans- 
ford, Brown, & Cocking, 1999). 

Metacognitive strategies can be taught. Since metacognition often 
takes the form of an internal dialogue, until recently it was assumed 
that individuals simply had to develop this capability on their own. 
Today, however, new research tools are available that allow researchers 
to closely monitor what experts do and think as they work. It is now 
clear that the kinds of metacognitive skills that experts in a given sub- 
ject area use can be identified. New approaches to teaching these 
metacognitive strategies to novices are being developed every day 
(Bransford, Brown, & Cocking, 1999). 



The EFF Framework is designed to help students develop their 
metacognitive awareness by making the metacognitive aspects of 
learning transparent. Metacognitive skills associated with “good 
performance” are written into the Components of Performance for 
the EFF Standards. Students begin their work on a Standard by 
identifying what they already know about the topic. Next, they 
closely examine what good performance looks like. They develop 
their own learning checklists so they can plan and monitor their 
understanding as they are practicing new skills. They also 
learn how to evaluate how well they are performing in rela- 
tion to the Standard through the use of scoring guides they 
help to develop, teacher interviews, portfolios, and other 
learner-centered assessment tools. (See Research to Practice 
Note 2 for more information about metacognitive strategies.) 
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Program Practices That Support Purposeful Teaching and Learning 



Results That Matter: An Approach to Program Quality Using Equipped for the Future 
(Bingman & Stein, 2001) provides a vision for program-level system reform 
(referred to as the EFF Quality Model). The EFF Quality Model identifies Program 
Practices that reflect the theoretical foundations of EFF and provides a guidepost 
by which administrators, teachers, students, and communities can assess their 
implementation of the EFF Framework. As you reflect on the examples below, 
think about how your program might answer the questions “What does it mean to 
practice EFF?” and “What does EFF implementation look like in action?” 



EXAMPLE 1: 

Students use the EFF 
Framework to clarify 
their purposes for 
learning and to identify 
strengths and gaps in 
the skills and knowledge 
necessary to achieve 
their purposes and 
goals. 



D uring their intake process, adult ESOL learners often identify goals related to 
communicating in their family and community life. A common concern is using 
English when one “goes to the doctor.” Alysan Croydan, a teacher from the Refugee 
Women’s Alliance of Seattle, Washington, describes how the EFF Framework helped 
her to delve more deeply to understand students’ specific needs related to obtaining 
medical care. 

After Alysan’s students expressed the need to improve their ability to make appoint- 
ments with doctors, they began working on the EFF Standard Speak So Others Can 
Understand by practicing dialogues in English. After they finished the activity, Alysan 
was surprised when many students continued to name making appointments with doc- 
tors as their goal. Using the EFF Framework, she asked them to reflect on what was still 
difficult about making an appointment. It became clear that the real issue was not just 
making an appointment, but finding a time that would fit their work and childcare 
schedules. Going beyond the scripted dialogues, she began teaching them how to nego- 
tiate an appointment time and troubleshoot scheduling problems. 



This activity might lead Alysan and her students toward work on other EFF Standards. 
For example, they might use the EFF Standard Solve Problems and Make Decisions to 
improve their skills related to addressing scheduling problems. The Components of 
Performance for this Standard guide students to anticipate or identify problems; use 
information from diverse sources to arrive at a clearer understanding of the problem 
and its root causes; generate alternative solutions; and select an alternative that is most 
appropriate to goal, context, and available resources. Depending on their needs, 
the students might decide to move on to another EFF Standard, such as Advocate 

Fov reflection. . . and Influence, in order to work with their employers to allow for time off for 

T , , _ rr, r doctor visits. 

Look at the Components of Performance 

for each of the EFF Standards. 

■ What is the common thread of purposeful A close investigation of students’ complex purposes for learning using the 

learning in each one ? EFF Standards not only helped Alysan to better understand their needs, 

• How do the other components in each but also helped to clarify what needed to be learned. As what they needed to 

Standard build on the initial naming learn became clear, they were better able to judge their own progress toward 
of a real-life purpose or goal? reaching their goals. 
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EXAMPLE 2: 

Teachers use the 
EFF Framework to 
structure a goal-setting/ 
needs-assessment 
dialogue with students. 



G ail Hemsoth of Lane Community College in Eugene, Oregon, teaches adults in a 
local welfare to work program. After a discussion of the EFF Standards, these stu- 
dents decided that they wanted to think more about how to develop skills related to the 
Standard Speak So Others Can Understand. With Gail’s help in simplifying the language, 
they looked over the Components of Performance for this Standard: Determine the 
purpose for communicating; Organize and relay information to effectively serve the 
purpose, context, and listener; Pay attention to conventions of oral English communi- 
cation, including grammar, word choice, register, pace, and gesture in order to mini- 
mize barriers to listener’s comprehension; and Use multiple strategies to monitor the 
effectiveness of the communication. Then they worked in small groups to brainstorm 
examples of tasks from their own lives where they needed to use these skills. Overnight, 
Gail compiled the task examples from all the groups and the next day asked the groups 
to rank the tasks in order of difficulty. When they finished, each learner established his 
or her own goals related to the Standard. For example, one woman was going to be 
starting a job soon in which she would have to give presentations on domestic violence. 
Another woman wanted to speak more confidently at job interviews. 



For reflection. . . 

• How might the experiences of 
learners in Gail’s program have been 
different if she had simply assigned 
them the task of making presentations 
about jobs on the first day of class rather 
than asking them to decide on the 
activity? 

• How do you help learners to come to 
consensus about learning activities 
they can work on together? 



Once the group became clear on their own personal goals, they began thinking about 
how they could plan a common activity that would allow them all to work together on 
the Standard. All the students recognized that they needed more practice in speaking in 
front of a group. They decided that since almost everyone also needed to spend more 
time identifying jobs that would interest them, they would combine these two goals by 
researching interesting jobs and then making a presentation about what they 
learned to the whole class. 



Together they discussed what a good oral presentation to the class would look 
like. They discussed how to tailor their talk to their audience, how they would 
know if they were being understood, and whether or not to use the overhead 
projector or handouts. Then they created a checklist that the audience would 
use to evaluate the presentation. After everyone was done, they reflected individ- 
ually and as a group about what they had learned and what they would do next in 
order to become more expert at speaking-related tasks. 



EXAMPLE 3: 

Teachers use EFF as 
a common language 
to discuss how their 
instructional practice 
supports attainment of 
student goals and 
purposes. 



P rogram administrator Jane Knight of Knox County Adult Literacy Program in 
Knoxville, Tennessee, describes how she and the teachers in her program used the 
framework as a common language to solve problems. Over Christmas break, Jane was 
able to set aside four weeks for the team to learn about EFF and create an action plan. At 
first, teachers felt a bit overwhelmed and worried about what changes would be required, 
but over time they learned to use the EFF tools and became more confident. Bringing on 
new teachers, however, was sometimes harder. For example, in one class, made up of ex- 
offenders, there had been a lot of teacher turnover. The students came to Jane with a list 
of complaints they felt the new teacher, who had been in the class for only four days, had 
not solved. Jane and another teacher experienced with EFF decided to visit the class. They 
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encouraged the students to use the Standard Solve Problems and Make Decisions to work 
on their concerns. Students began to speak up about how difficult it was for them to make 
it to the bus on time because they had to take their children to school. If they missed the 
bus, the next one did not come until an hour later. As ex-offenders, these learners were 
required to report their attendance to the court system. Being late resulted in a penalty. As 
they worked together to generate alternatives, the students were able to see how, if every- 
one compromised a little, an effective solution could be found. This was a pivotal event 
for the students, the new teacher, and the program. The students began to see that they 
For reflection. . . were part of a programwide community with their own set of responsibilities. The 

• How could your program use EPF new teacher was able to see how EFF worked in action. Growing out of this expe- 
as a common language to plan and rience, the program decided to hold monthly Town Meetings where the students 

discuss your educational practices? and staff could discuss programwide issues. 



Glossary 

Cognitive science: The study of thinking and learning, 
currently being contributed to by researchers in a wide variety 
of disciplinary and multidisciplinary fields from developmental 
psychology to medicine. (See Bransford, Brown, & Cocking, 
1999.) 

Common Activities: The term EFF uses to refer to those 
activities that adults perform in all three roles (worker, family 
member, community member). The EFF team identified the 
13 Common Activities by looking across the Broad Areas of 
Responsibility, the Key Activities, and the Role Indicators for 
each Role Map. (See Stein, 2000, p. 14; Merrifield, 2000, 
pp. 33-34.) 

Content Standards: The term used in a variety of fields to 
describe what individuals need to know and be able to do for a 
particular purpose. In EFF, the 16 Content Standards identify 
what adults need to know and be able to do in order to meet 
their goals for learning and to be effective in their adult roles. 
Each EFF Content Standard consists of the title of the standard 
and the Components of Performance for that standard. (See 
EFF Standards, Stein, 2000, pp. 19-20.) 

EFF Quality Model: A vision of what system reform at the 
program level looks like using EFF Standards. The EFF tools, 
foundational theory and research, expected program practices, 
and predicted short- and long-term outcomes are presented 
and explained in the publication Results That Matter: An 
Approach to Program Quality Using Equipped for the Future 
(Bingman & Stein, 2001). Ordering and downloading 
information can be found at http://www.nifl.gov/lincs/ 
collections/eff/eff_publications.html. 



Mental model: An individual’s existing understanding and 
interpretation of a given concept, which is formed and 
reformed on the basis of experiences, beliefs, values, sociocul- 
tural histories, and prior perceptions (Fambert & Walker, 1995, 
p. 1). Our mental models (or schemas) affect how we interpret 
new concepts and events. 

Metacognition: The capacity to reflect on one’s own thinking 
(Greeno, Resnick, & Collins, 1997, p. 19). Metacognitive strate- 
gies include monitoring our thinking and understanding while 
we work, checking to see if what we are learning is consistent 
with what we already know, and making analogies that will 
help our understanding (Bransford, Brown, & Cocking, 1999). 

National Education Goal 6: One of the National Education 
Goals identified by the 50 governors and President George 
Bush at an education summit in 1989 and later enacted by 
Congress as part of the Goals 2000 Act. Goal 6 is the only goal 
directly related to adult learning and is often referred to as the 
Adult Fiteracy and Fifelong Learning Goal. A congressional 
mandate to measure progress toward Goal 6 was the impetus 
for the development of EFF. (See Stein, 2000, pp. 5-7.) 

Prior knowledge: The knowledge one already has about a 
given topic. Prior knowledge may include accurate as well as 
inaccurate preconceptions about how the world works. Activat- 
ing learners’ prior knowledge about a topic and involving them 
in revising or building on it is an essential step in effective 
learning. (See Hartman, 2001.) 

Purposeful approach to education: Teaching and learning that 
is designed specifically around the goals and purposes of 
students in their real-life roles as family members, community 
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members, and workers. A purposeful approach assumes 
intentionality, explicitness, and transparency in the learning 
environment. The EFF Standards contribute to purposeful 
learning because they make explicit and transparent the skills 
adults need to meet their goals and purposes. (See Bingman 
8c Stein, 2001; Merrifield, 2000, p. 9.) 

Purposes for Learning: The four fundamental purposes that 
adults offer as reasons for furthering their literacy education. 
The four Purposes for Learning are (1) Access and Orientation, 
(2) Voice, (3) Independent Action, and (4) Bridge to the 
Future. These purposes drive learning across the different con- 
texts of adult life and capture the social and cultural signifi- 
cance of learners’ specific, individual goals (Merrifield, 2000, 
pp. 13-17). (See Stein, 1995; Stein, 2000, pp. 5-6.) 



Role Map: A publicly agreed to, explicit, consensus depiction 
of the adult roles of worker, parent/ family member, and 
citizen/community member. For each adult role, the Role Map 
provides definitions of the Broad Areas of Responsibility, Key 
Activities, and Role Indicators, which describe, not prescribe, 
effective performance in the role. (See Stein, 2000, pp. 8-13.) 

Transparent approach: An approach to teaching and learning 
in which the goals and purposes of learning, what will be 
learned, and what good performance looks like are clear and 
explicit to students, teachers, administrators, and other stake- 
holders. The EFF Standards are important in this approach 
because they clearly define the skills adults need to meet their 
goals and purposes. 
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HANDOUT 2.7 

A Contextualized Approach to Curriculum and Instruction: 
Background and Case in Point 3 

Background: 

In effective adult teaching and learning environments, research shows that attention to context — the 
environments, conditions, and circumstances in which learners exist — helps students apply knowledge, skills 
and processes across contexts, and that the learning context itself contributes to the learning process. For 
additional background information, see EFF Research to Practice Note 3. Understanding learners’ contexts is 
important for identifying EFF Standards that would help learners achieve their goals. It is also important to 
understand the many contexts of learners’ lives when focusing and designing the learning activity. Learning 
activities that directly connect to learners’ lives are more effective. 

Building expertise is a complex developmental process in which new knowledge is built on prior knowledge. 
Students need cognitve and metacognitive strategies to learn and apply new information. For additional 
information on building expertise, see EFF Research to Practice Note 2. The EFF framework, particularly the 
EFF Standards and the Teaching/Learning Cycle, gives teachers a structure for designing learning activities 
that address the issues arising from the contexts of students’ real lives while also building strategies and 
basic skills. 

Administrators can support contextualized teaching and learning by clarifying both their programs and 
students’ contexts. Awareness of these contexts might mean that administrators foster alignment of program 
goals with student goals and student contexts, budget for materials relevant to the teaching and learning 
contexts of students and the program, plan and support staff development activities that focus on 
contextualized curriculum development, and design and monitor assessment and evaluation processes 
based on contextualized instruction. 

Administrators can play an important role in supporting activities that extend learning to different contexts 
outside of the program. To do this, it might mean that administrators also network with and disseminate 
program goals and activities to community partners such as employers, other educational institutions, 
community-based organizations, and social service agencies. 

The question for the team: 

How will this agency help students build expertise through a contextualized approach 
to curriculum and instruction? 



Case in Point: 

Kenesshia, one of Ms. Franklin’ s GED students, has recently moved into her own apartment. She has 
been working on math skills in preparation for the GED test. Ms. Franklin is interested in knowing how 
Kenesshia is applying the math she is studying in class to her new situation, managing her own apart- 
ment and finances. One day Ms. Franklin asked Kenesshia how things were going with the apartment. 
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Kenesshia, despondent, said, “I have a problem; I don’t have any nice furniture. The apartment looks so 
empty.” When Ms. Franklin asked what she was planning to do about the problem, Kenesshia responded 
that she was thinking about visiting the new rent-to-own store in her neighborhood. Ms. Franklin sug- 
gested to Kenesshia that she “research” the situation before making any decisions and commitments. 

Ms. Franklin knew that Kenesshia could use both the Use Math to Solve Problems and the Plan Standard 
to help her make a decision about getting furniture from the rent-to-own store. She asked Kenesshia if 
she would give a report to the class on what she discovered when researching the rent-to-own plans. 

Ms. Franklin and the rest of the class were stunned by the report Kenesshia gave on the rent-to-own 
plans. “It’s a rip-off,” said Kenesshia. When Kenesshia figured out the cost of renting a living room set, 
she realized she would have paid for it three times over. The television and stereo set would have cost her 
four times as much through the rent-to-own plan. Then Kenesshia talked about the math skills she used 
to analyze the rent-to-own situation. She gave a long list that included whole number math, fractions, 
and even some algebra. Ms. Franklin and the rest of the students were proud of Kenesshia, thanked her 
for her report, and brainstormed with her some alternatives for getting new furniture. 

Ms. Franklin and Kenesshia could align and assess the learning based on the Standards’ components and 
how well Kenesshia could transfer skills to other contexts and in other roles. 
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EFF RESEARCH TO PRACTICE NOTE 




EFF Research Principle: 

An Approach to Teaching and Learning 

That Builds Expertise By Marilyn K. Gillespie 




What Do We Mean by an Approach That Builds Expertise? 



T he conceptual framework for Equipped for the Future (EFF) is based 
in part on a theory of knowing and learning known as construc- 
tivism. This theory conceives of learning as an active process of 
knowledge construction. Learners use their prior knowledge and 
experience to shape meaning and acquire new knowledge. 

Within this approach, learning is viewed as a process of activating our prior 
knowledge related to a topic we want to learn about; questioning, interpreting, 
analyzing, and processing new information and concepts in light of our past expe- 
riences; using this information and our thinking processes to monitor, develop, 
and alter our understanding; and integrating our current experiences with our 
past experiences (see Fosnot, 1992; Lambert & Walker, 1995; Mayer, 1998; 
Larochelle, Bednarz, & Garrison, 1998; Duffy & Jonassen, 1992; Brooks & Brooks, 
1993; Cromley, 2000). Work in this area is closely linked to cognitive science 
research related to the development of expertise (see Bransford, Brown, & Cock- 
ing, 1999; Glaser, 1992). 

This Research to Practice Note will describe how research findings related to con- 
structivism have been applied to the development of the EFF Framework and 
the EFF Continuum of Performance, a multi-dimensional developmental 
description of performance that serves as a foundation for EFF-based instruc- 
tion and assessment of learner progress. Among the key findings addressed are 
the following: 

• Acquiring expertise is a complex developmental process in which new knowl- 
edge is built on prior knowledge. 

• To develop expertise, learners need a richly structured knowledge base. They 
need to learn cognitive and metacognitive strategies for using and applying 
new information. 

• Scaffolding instruction helps learners to develop their fluency, independence, 
and range of performance as they move along a developmental continuum 
from novice to expert. 



Research Findings on Building Expertise 



Building expertise is a complex developmental process. 

Most of us were taught in accordance with a “knowledge acquisition” model of 
learning (Mayer, 1998). In school, we were required to accumulate knowledge 
about a subject in separate “bits” of information. The order in which we learned 



The EFF publication Results 
That Matter: An EFF Approach 
to Quality presents five key prin- 
ciples that reflect the theoretical 
foundations of EFF. Program 
practices that support these 
principles provide guideposts 
by which programs, teachers, 
students, and their communities 
can assess their implementation 
of the EFF Framework. They 
help practitioners to better 
answer the questions “What 
does it mean to practice EFF?” 
and “What does EFF implemen- 
tation look like in action?” 

These Research to Practice 
Notes will help you to: 

• identify the research basis for 
the principles; 

• learn key concepts and terms 
associated with the principles; 

• see examples of how other 
programs have implemented 
the program practices; 

• reflect on how you and your 
program can implement the 
program practices. 
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Rather than seeing learning as 
the rote acquisition of knowledge, 
researchers have come to see 
learning as a process of sense- 
making. Learners do not simply 
absorb, passively receive, or 
record objective knowledge that 
is “out there.” They actively 
construct and interpret knowledge 
by integrating new information 
and experiences into what they 
already know. 

— Mayer (1 998) 



these facts was tightly sequenced into a hierarchy of behavioral objectives. After 
enough drills or practices, we were tested to ensure we had mastered these objec- 
tives before we proceeded to the next objectives (Shepard, 2000). 

Toward a model of “knowledge construction.” Although learning content 
knowledge is important to developing expertise, new cognitive research has 
revealed that it is not enough to fully prepare learners to use that knowledge in 
the real world. Over the past two decades, research studies have closely examined 
how experts in a growing number of fields (including math, science, music, 
reading, and history) learn and apply what they have learned. There is now 
strong evidence that experts do not just know more facts. They are not “smarter,” 
nor do they necessarily have better memories than other people. Rather, they 
have developed a more complex, richly structured knowledge base related to 
their field. (For a review of research on the development of expertise, see Brans- 
ford, Brown, & Cocking, 1999; Donovan, Bransford, & Pellegrino, 1999; and Pel- 
legrino, Chudowsky, & Glaser, 2001.) 



How experts acquire and use knowledge. Experts with a strong knowledge base 
are able to (1) extract a level of meaning from content information that is not 
apparent to novices by structuring what they know into meaningful patterns 
and relationships, (2) organize their knowledge around core concepts and big 
ideas, (3) apply cognitive strategies to select and remember information that is 
relevant and eliminate what is unimportant, and (4) use metacognitive strate- 
gies to “conditionalize” their knowledge by knowing when certain concepts are 
useful and fluently retrieving the information necessary to solve a problem at 
hand. This complex knowledge base extends experts’ ability to use what they 
know and to transfer knowledge from one problem or context to another (von 
Glasersfeld, 1987). 



For reflection. 



Can you think of areas in your life as a 
parent, family member, worker, or community 
member where you might be considered an expert ? 
What kinds of knowledge, skills, and strategies have 
you developed over time? 

How do you organize what you know around 
“ big-picture ” ideas? 

How did you become more fluent and 
independent in performing this role? 



Adult performance along a developmental continuum. The EFF Assessment 
Consortium has drawn on this understanding of the development of expertise 
to define and develop a continuum of performance that shows how adults grow 
and learn throughout their lives, constructing new knowledge, skills, and abili- 
ties that allow them to respond flexibly to change. The EFF Continuum of Per- 
formance enables us to see how competence in a Standard develops along 
multiple dimensions as learners move from the novice to the expert 
level. Four key Dimensions of Performance distinguish perfor- 



mance along this developmental continuum for each of the EFF 
Standards: the Knowledge Base, Fluency, Independence, and 
Range dimensions. Understanding these Dimensions of Perfor- 
mance helps teachers to plan instruction, as well as to determine 
how well students are able to use the skills and knowledge associ- 
ated with each Standard. 
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A richly structured knowledge base includes knowing how to use 
and apply cognitive and metacognitive strategies. 

Our knowledge of what strategies are and how they work in the development 
of expertise comes out of a strand of cognitive research called information 
processing (Hartman, 2001; Pressley & Woloshyn, 1995; Greeno, Resnick, & 
Collins, 1997). This research on how the brain processes information has 
shown that new content knowledge we acquire is first stored in our short- 
term memory. However, our short-term memory has only a limited capacity 
to hold information. We have to process this information in some way or it 
will fade quickly. Learning strategies are defined as any behavior, thought, or 
action that allows learners to process information so that it can more effi- 
ciently be stored in and later retrieved from long-term memory (Weinstein & 
Hume, 1998). 

Experts as good strategy users. Until recently, we have known little about how 
these strategic processes work since they are often used automatically and uncon- 
sciously by experts. However, through closely monitored research asking experts to 
“think aloud” as they work, we have begun to see how powerfully learning strate- 
gies influence expert learning. We now know that these strategies can be explicitly 
identified and taught to more novice learners (Pressley & Woloshyn, 1995). 

Cognitive and metacognitive strategies. Learning strategies can be divided into 
two basic types. Cognitive strategies help us to remember and organize content 
information. For example, when we read, we might apply a cognitive strategy to 
skim the title, pictures, and headings of a text to get the gist of what we will read. 
We might take notes to help us remember the main points. A good reader will 
also know when it is possible to skip over sections of text and when it is impor- 
tant to read every word carefully. When learning a large number of facts, a good 
strategic learner will “study smarter” by working to understand the “big picture” 
and then dividing the facts into categories through a classification scheme, dia- 
gram, or outline. 

Metacognitive strategies. Metacognitive strategies consist of knowledge about 
strategies and about one’s own thinking processes. They are the “executive man- 
agers” of knowledge and include planning, monitoring, evaluating, and revising 
one’s own thinking (Hartman, 2001). Good metacognitive strategy users engage in 
an ongoing process of identifying what their prior knowledge of a topic is, what 
they don’t know, and what they need to learn. Metacognitive strategies enable 
learners to plan and self- regulate their work and to judge under what conditions 
to apply which cognitive strategies. 

Strategy acquisition and EFF. Each of the EFF Content Standards identifies 
strategies as key components of using the Standard to carry out tasks in every- 
day life. In addition, cognitive and metacognitive strategies are also being 
explicitly identified and described as an integral part of the Knowledge Base 
dimension of the EFF Performance Continuum for each Standard. What the 



Since I began developing 
performance tasks for EFF, I look 
at the student’s learning process 
differently. . .1 am much more 
observant because I have to describe 
the strategies that students use to 
apply their knowledge to complete 
the task. I have learned to watch 
more carefully and then work to 
put my observations into words. 

— Nancy Gepke, Tacoma, Washington 
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For reflection. . . 

• How does your program currently emphasize “knowledge 
construction” as well as “knowledge acquisition”? 

• What are some ways your program includes the teaching of 
cognitive and metacognitive strategies as part of instruction? 

• Where can you go to get more information about learning 
strategies and how to teach them? 



strategies are changes as one moves along the developmental 
continuum from novice toward expert. For example, within 
the Standard Read With Understanding, at the novice level 
a reader might be expected only to be able to restate 
what was read. As readers move along the continuum, 
higher-order processing skills such as synthesis and 
analysis are required. 



DIMENSIONS 
OF PERFORMANCE 


Assessing Learner Performance: 

EFF Assessment Consortium (2002) EFF/NRS Data Collection Project Interim Report 


Knowledge Base 

What do learners know? 


1 . What vocabulary do learners have related to the skill? Related to the subject area? 

2. What content knowledge do learners have related to the skill? Related to the subject area? 

3. What strategies do learners have for organizing and applying content knowledge? 

• Can learners recognize or create new relationships or connections? 

• Can learners identify information that is important to the task/problem? 

• Understand when information or concepts apply? 


Fluency 


How fluently can learners perform? 

• How much effort is required? 

• How consistently do learners start and finish, getting to the desired outcome? 

• How well are barriers controlled or overcome? 


Independence 


How independently can learners perform? 

• How much help is needed from others? 

• How much initiative is shown in getting started? 

• How often do learners generate their own strategies to complete the task? 


Range 


1 . What kinds of tasks do learners carry out? 

• How complex is the task? 

• How many different kinds of tasks can learners perform? 

2. In what contexts can learners perform? 

• In what kinds of contexts? 

• In how many different situations can learners perform? 



Scaffolding helps learners to develop their fluency, independence, 
and range of performance as they move from novice to expert. 

Teaching to the zone of proximal development. In addition to developing learn- 
ers’ knowledge base, teachers using an EFF approach also work with learners to 
develop their fluency, independence, and range of performance. In helping learners 
to move along the continuum in relationship to these dimensions, EFF teachers 
have drawn on the work of another thinker whose work is closely associated with 
constructivist theory: the Russian psychologist Lev Vygotsky (Vygotsky, 1978; 
Dixon-Krauss, 1996; Wertsch, 1991). Vygotsky found that new capabilities in a 
novice learner are first developed during collaboration with teachers or more com- 
petent peers and then internalized to become part of the individual’s mental model 
of the world. Vygotsky called the distance between what an individual can accom- 
plish independently and what he or she can accomplish with the help of someone 
who is more competent the zone of proximal development. The role of education, 
he believed, is to provide learners with experiences that are within their zone of 
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proximal development — with tasks that are slightly above their level of indepen- 
dent functioning yet can be accomplished with sensitive guidance. 

Scaffolding instruction. Vygotsky viewed the social environment as a necessary 
scaffold or support system that allows a learner to move forward and continue to 
build new competencies, just as scaffolding is used by a painter to reach parts of a 
house that would otherwise be out of reach (Berk & Winsler, 1995). In the process 
of jointly performing a task, the teacher or a more skilled peer can point out links 
between the task and ones the learner already knows, helping the learner to stretch 
his or her understanding into the next development level. Within EFF classrooms 
that use this approach, the teacher’s role is to first structure the task and the learn- 
ing environment so that the demands on the learner are at an appropriately chal- 
lenging level and then to continually adjust the 
For reflection. . . amount of intervention and the range of 

• In what ways does your program tasks to the learner’s level of in depen - 

design activities to scaffold instruction? dence and fluency. In this way, teachers 

• How might scaffolding be applied in a can use the developmental continuum as 
classroom of multi-level learners? a guide for learning and instruction. 



Building Expertise in Your Program 



Results That Matter: An Approach to Program Quality Using Equipped for the Future 
(Bingman & Stein, 2001) provides a vision for program-level system reform 
(referred to as the EFF Quality Model). The EFF Quality Model identifies Program 
Practices that reflect the theoretical foundations of EFF and provides a guidepost 
by which administrators, teachers, students, and communities can assess their 
implementation of the EFF Framework. As you reflect on the examples below, 
think about how your program might answer the questions “What does it mean to 
practice EFF?” and “What does EFF implementation look like in action?” 



EXAMPLE 1: 

Teachers and students 
use the EFF Framework 
to examine prior knowl- 
edge, to construct new 
knowledge in light of 
their past experiences, 
and to use this informa- 
tion and their thinking 
processes to monitor, 
develop, and alter their 
understanding. 



J enny Bolte is a teacher in a worksite -based program in Virginia. One of her stu- 
dents, a carpenter’s helper named Donnie, initially came to the program to work on 
his reading skills. Before learning how to use the EFF Framework, Jenny might simply 
have tested Donnie to find out his reading level and then found workplace literacy 
materials he could read, accompanied by comprehension questions he could answer. 
Instead, Jenny started off by introducing Donnie to the Worker Role Map and Com- 
mon Activities. The language of the Worker Role Map helped Donnie begin to talk 
about a pressing and immediate problem he was facing. His supervisor wanted him 
to go for a promotion to First-Class Carpenter. Donnie was not sure if he wanted to 
do so or not. He didn’t know what was involved and wasn’t sure whether his skills 
were up to the task. 

Jenny introduced Donnie to the EFF Content Standards. It became clear to both of 
them that the Standard Learn Through Research might give Donnie the tools to help 
him make a decision about the promotion. The first component of the Standard 
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Learn through Research 

• Pose a question to be 
answered or make a 
prediction about objects 
or events. 

• Use multiple lines of 
inquiry to collect 
information. 

• Organize, evaluate, 
analyze, and interpret. 



helped him to think about what he already knew about being a First-Class Carpenter 
and what he needed to find out. The second component encouraged him to use multi- 
ple lines of inquiry to collect information. He decided first to talk with people at work. 
Jenny recommended Duke West, the HR Maintenance Training Coordinator. Donnie 
agreed. “Who else?” she questioned. After some thinking, he said, “How about Sid?” 
They decided his co-worker Sid would be a “good one to ask” as well. Jenny suggested 
they also try to get a copy of the training manual for the job so they could analyze what 
kinds of reading and writing the job required. Once all the information was collected, 
Jenny would help Donnie with the third component: organizing, evaluating, analyz- 
ing, and interpreting what he had learned in order to make his decision about whether 
to go for the job. 



In her work with Donnie, Jenny used a constructivist approach to teaching and learning 
that allowed Donnie to use his prior knowledge to construct new knowledge related to 
a real-world purpose. Instead of using a textbook, they had arrived at the content for 
instruction through an interaction between the EFF Framework and Donnie’s under- 
standing of his immediate needs. Since what Donnie wanted to learn was rooted in a 
For reflection. . . real-life context, a much wider repertoire of skills came into play. To complete this 

• How is Jenny’s approach to teaching task ’ Donnie needed to develo P his rcadin 8 and writing skills, but he was doing 



and learning similar to or different from 
your own? 

How might Donnie have changed his view 
of himself as a learner as a result of this 
experience? 

What kinds of program practices support 
or hinder a constructivist approach to 



so in light of real-world needs and applications. His reading material was the 
training manual for his job. His writing tasks included drafting and revising 
questions for interviews and making a “pros and cons” chart to analyze his 
findings. When they were done, Jenny would help Donnie reflect on what 
he had accomplished: to look more closely at the knowledge, skills, and 
learning strategies he had used to complete this task; to think about how he 
might transfer what he had learned to other parts of his life; and to determine 



instruction? 



what he wanted to learn next. 



EXAMPLE 2: 

Teachers and students 
use the EFF Framework to 
identify, reflect on, and revise 
their own mental models of 
adult role performance. 

For reflection. . . 

• How might the learning outcomes for 
E. W. have been different if Marty had focused 
on reading skills alone, without building in a 
process for her to identify and revise her mental 
model of herself as a learner? 

• How have you worked with students who have 
been able to change their mental models of adult 
role performance? What kinds of teaching and 
learning made this change possible? 



M any adult learners come to the classroom with existing mental models of them- 
selves that create a barrier to learning. An example can be seen in the story 
Marty Duncan told of her work with E. W., who came to a Vermont literacy program 
a year after having lost her husband of 30 years. For years, E. W. had depended on her 
husband for assistance with reading and had held a lifelong belief that her own 
“thick-headedness” was the reason she hadn’t learned how to read. She lacked content 
knowledge with respect to reading, but she also was not able to use what she did 
know because she lacked strategies for when and how to apply her knowledge of 
reading. Marty scaffolded E. W.’s learning by connecting reading and writing 
to something she already knew about: grocery lists. At the same time, she 
also helped E. W. to examine her knowledge of herself as a learner and 
to develop metacognitive strategies for overcoming her internal barri- 
ers to comprehension. Slowly, as E. W. learned how to reflect on her 
success at remembering what she learned, she also became a more 
independent learner more and more willing to suggest next steps. By 
focusing on reading skills along with metacognitive strategies to 
revise and monitor her own learning, E. W. was able to overcome her 
internal barriers to learning. 
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EXAMPLE 3: 

Teachers use the EFF 
Framework to help 
learners develop their 
fluency, independence, 
and range of perfor- 
mance as they move 
from novice to expert. 



For reflection. . . 

• This teacher used a performance-based 
approach to teaching ESOL learners. 

How might the use of performance-based 
instruction help your students to develop 
their fluency, independence, and range 
of performance on tasks they need 



E FF Trainer Andy Nash describes an EFF classroom in Chula Vista, California, where 
a group of adult English language learners expressed a need to find out more about 
affordable eye care. Their teacher, Judy Wurtz, turned to two Standards, Learn Through 
Research and Speak So Others Can Understand, to guide the development of these skills 
in a project where students researched available low-cost eye exams and glasses. 

Since these students had limited English language skills, the teacher broke the learning 
activities down into a series of discrete steps. She used guided language scripts and 
worksheets to scaffold each step. For example, in the first step, the learners practiced 
using the yellow pages and then worked in teams to find telephone numbers of eye care 
centers in a phone book. Next, the learners developed and practiced scripts for what to 
say when they phoned the eye care center. They then made phone calls. Judy helped 
learners to develop a simple chart to keep track of the information they got from the 
calls. The class even attended a community event where they made contact with an 
agency participating in a national project to provide eye care for students. As 
they worked together, Judy was helping her students to gain the independence 
and fluency they needed to perform this task on their own. That many suc- 
ceeded is evidenced by Judy’s report in her teaching log that eight students 
received eye appointments and most of them got glasses. As a next step, Judy 
might consider having learners think about expanding their range of perfor- 
mance by using similar skills to make other kinds of appointments in their roles 



for everyday life ? 



as parents, family members, workers, or citizens. 



Glossary 

Cognitive strategies: Any behavior, thought, or action a learner 
engages in during learning that is intended to influence the 
acquisition, storage in memory, integration, or availability for 
future use of new knowledge and skills. (See Weinstein & 

Flume, 1998, p. 12; Pressley & Woloshyn, 1995.) 

Constructivism: A theory of learning and knowing that holds 
that learning is an active process of knowledge construction in 
which learners build on prior knowledge and experience to 
shape meaning and construct new knowledge. (See Lambert & 
Walker, 1995.) 

Continuum of Performance: A multidimensional, developmen- 
tal description of performance on an EFF Standard ranging 
from the novice level to the expert level. The continuum is built 
around the four Dimensions of Performance, and performance 
levels are defined by identifying key features of performance at 
various points along the continuum. (See Stein, 2000, pp. 58-59.) 

Dimensions of Performance: The theoretical foundation, based 
in cognitive science, on which the EFF Continuum of Perfor- 
mance for each skill is built. The Dimensions of Performance 
identify developmental differences in performance on the EFF 



Standards related to four areas: (1) structure of the knowledge 
base, (2) fluency of performance, (3) independence of perfor- 
mance, and (4) range of conditions for performance. (See Stein, 
2000, pp. 59-60; Bransford, Brown, & Cocking, 1999.) 

• Structure of the knowledge base: The organization and 
application of knowledge, skills, and strategies evidenced in 
performance. 

• Fluency of performance: The ease, fluidity, and/or automatic- 
ity evidenced in performance. 

• Independence of performance: The degree of initiative and 
self-reliance evidenced in performance. 

• Range of conditions for performance: The degree to which 
tasks and task contexts are familiar or unfamiliar to the learn- 
er, the extent to which tasks are structured (“scaffolded”) or 
unstructured, and the complexity of tasks. 

Metacognitive strategies: Metacognitive strategies consist of 
knowledge about strategies and about one’s own thinking 
processes. They are the “executive managers” of knowledge and 
include planning, monitoring, evaluating, and revising one’s 
own thinking. (See Bransford, Brown, & Cocking, 1999; 
Hartman, 2001.) 

Performance task: A learning activity with embedded assess- 
ment that meets learners’ purposes and addresses all compo- 
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nents of an EFF Standard. It informs all steps of the instructional 
planning, teaching, and assessment cycle. It addresses a real- 
world activity and can be analyzed according to the Dimensions 
of Performance. (See Ananda, 2000; McGuire, 2000.) 

Scaffolding: The structure and supports a teacher or more 
knowledgeable helper provides to allow a learner to perform a 
task he or she cannot yet perform independently. (See Vygotsky, 
1978; Dixon-Krauss, 1996; Wertsch, 1991.) 



Zone of proximal development: The distance between what an 
individual can accomplish independently and what he or she can 
accomplish with the help of someone who is more competent. 
This concept was first developed by Vygotsky (1978), who saw 
the role of education as to provide learners with experiences that 
are within their zone of proximal development — with tasks that 
are slightly above their level of independent functioning yet can 
be accomplished with sensitive guidance. (See also Dixon- 
Krauss, 1996; Wertsch, 1991; Berk 8c Winsler, 1995.) 
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EFF Research Principle: 

A Contextualized Approach 

to Curriculum and Instruction By Marilyn K. Gillespie 



What Do We Mean by a Contextualized Approach? 



A third key concept underlying Equipped for the Future (EFF) relates 
to its contextualized approach to curriculum and instruction. Instead 
of first teaching skills and knowledge separated from their context 
and hoping that learners will end up knowing how to transfer what 
they have learned to life outside the classroom, EFF teachers start 
with real-life contexts and weave these contexts into every stage of the teaching and 
learning process. Instruction and assessment are aimed directly at the skills and 
knowledge adults need to perform tasks they have identified as important and 
meaningful to them “right now” in their everyday lives. The focus is on the applica- 
tion rather than on the possession of basic skills and knowledge (Merrifield, 2000). 

The contextualized approach to instruction draws on the same body of cognitive 
research described in Research to Practice Notes 1 and 2. Of key importance for this 
principle is research on the transfer of learning. Research shows that learning 
transfers from one context to another more effectively when the learner under- 
stands not only the facts but also the “big picture” — the underlying principles, 
patterns, and relationships — that is acquired through the application of knowl- 
edge (Glaser, 1992; Bransford, Brown, & Cocking, 1999; Greeno, Resnick, & 
Collins, 1997). This contextualized approach is also based on the recognition that 
the development of expertise requires that a learner develop not only content but 
also procedural knowledge, such as the metacognitive awareness of when and how 
to apply what has been learned. This kind of knowledge can be acquired only 
through practice (Pressley & Woloshyn, 1995; Hartman, 2001). 

A contextualized approach to instruction also stresses the social nature of real- 
world activities (Wenger, 1998; Fave & Wenger, 1991), the value of building a learn- 
ing community within the classroom, and the importance of incidental learning 
that takes place when knowledge and skills are acquired within a social context. For 
example, when the skill of “filling out forms” grows out of an immediate real-world 
need of immigrant learners and is addressed in a community of learners, issues 
such as understanding the conditions under which filling out forms is necessary, 
when and how to call in an “expert” such as a lawyer, and the benefits and draw- 
backs of asking family members for assistance become part of the curriculum. This 
Research to Practice Note focuses on the following key assumptions: 

• Effective learning requires not only the acquisition but also the active applica- 
tion of knowledge, skills, and processes. 

• To encourage transfer to other contexts, effective learning requires the acquisi- 
tion of a complex knowledge base including content knowledge, skills, and cog- 
nitive and metacognitive strategies. 

• Learning is a function not only of the activity itself but also of the context and 
culture in which it takes place. 



The EFF publication Results 
That Matter: An EFF Approach 
to Quality presents five key prin- 
ciples that reflect the theoretical 
foundations of EFF. Program 
practices that support these 
principles provide guideposts 
by which programs, teachers, 
students, and their communities 
can assess their implementation 
of the EFF Framework. They 
help practitioners to better 
answer the questions “What 
does it mean to practice EFF?” 
and “What does EFF implemen- 
tation look like in action?” 

These Research to Practice 
Notes will help you to: 

• identify the research basis for 
the principles; 

• learn key concepts and terms 
associated with the principles; 

• see examples of how other 
programs have implemented 
the program practices; 

• reflect on how you and your 
program can implement the 
program practices. 



National Institute for Literacy 
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What Research Says about Contextualized 
Learning 



Effective learning requires not only the 
acquisition but the active application of 
knowledge, skills, and processes. 

In recent years, an accumulating body of research 
evidence has demonstrated that the acquisition of con- 
tent-related knowledge and skills alone is not sufficient 
for the development of expertise. To move from novice to 
expert levels of performance or competence, learners 
need to acquire both content knowledge and procedural 
knowledge related to when and how to apply what has 
been learned. (Procedural knowledge, including cognitive 
and metacognitive strategies, is described in Research to 
Practice Note 2.) Since procedural knowledge deals with 
the processing and application of skills, it can be learned 
only through action. For example, to develop procedural 
knowledge related to science, students need to simulate 
the kinds of activities real scientists do. To learn math, 
students need to think and act like mathematicians (von 
Glasersfeld, 1987; Glaser, 1992; Bransford, Brown, & 
Cocking, 1999). These same principles apply to workforce 
education, where nationwide studies (Secretary’s Com- 
mission on Achieving Necessary Skills, 1991; Carnevale & 
Porro, 1994) have confirmed that acquiring job-related 
content and basic academic skills is not enough to 
prepare adults and youth to be effective on the job. Just as 
important are interpersonal, decision-making, and plan- 
ning skills and the knowledge of when and how to apply 
these skills within the social context of the workplace. 
These skills require instructional approaches that focus 
on cooperative learning, apprenticeship models, and 
teamwork (Grubb, 1997; Kerka, 1997; National Associa- 
tion of Manufacturers, 2001). 

Within adult education, the concept of contextualized learning is not new. Sticht 
(1997) describes how as early as the 1940s, the armed forces had begun to contex- 
tualize instruction to reflect the everyday life experiences of soldiers. Functional 
context instruction (Sticht et al., 1974), instruction based on learners’ immediate 
needs and “life skills” (Knowles, 1980), and the importance of teaching for transfer 
(Mikulecky, Albers, & Peers, 1994) have been advocated by adult educators since 
the early 1970s. 

The EFF development team drew on this knowledge base to develop its field 
research process (Stein, 2000). Adult learners, teachers, program directors, and 
content experts from around the United States engaged in an intensive process of 
mapping what adults commonly need to know and do to be effective in everyday 



Designs for Learning Environments 

The authors of How People Learn: Brain, Mind, 
Experience, and School (Bransford, Brown, & Cocking, 
1999) identified the following as key components that 
work together to make up a contextualized and mutually 
supportive learning system: 



■ Learner-centered 
environments. Effective 
learning begins with what 
learners bring to the environ- 
ment; this includes cultural 
practices and beliefs, as well 
as knowledge of academic 
content. Evidence shows 
that learners use their cur- 
rent knowledge to construct 
new knowledge and what 
they know and believe at the 
moment affects how they 
interpret new information. 

■ Knowledge-centered 
environments. The ability to 
think and solve problems 
requires that knowledge of a 
subject area be accessible 
and linked to current 
understanding. Designs for 
subject area study should 
help students learn with 
understanding instead of 
promoting the acquisition of 
disconnected sets of facts 
and skills. 



■ Assessment-centered 
environments. Students’ 
thinking must be made 
visible, and feedback must 
be provided on an ongoing 
basis to give them the 
opportunity to revise and 
improve the quality of their 
thinking and understanding. 
The kinds of assessment 
chosen should reflect their 
learning goals. 

■ Community-centered 
environments. The learning 
environment should promote 
a sense of community. 
Classroom norms should 
encourage students to 
learn from one another 

and support one another’s 
improvement. Learning in 
school should be connected 
with outside learning 
activities. 
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life. The EFF Role Maps, Common Activities, and Content Standards that grew 
out of this process provide teachers with the tools through which to help learners 
identify their broad purposes and immediate needs. Once these needs have been 

identified, teachers can work backwards to deter - 
Fov reflection. . . mine the knowledge, skills, and strate- 

• Think about a situation where you had a 8^ es learners need to accomplish 

chance to learn through the active application *h e tasks. Only then do they 

of knowledge and skills. What difference did reach the step of developing 

it make to what and how you learned? learning activities. 

To encourage transfer to other contexts, effective learning 
requires acquiring a knowledge base of content knowledge, 
skills, and strategies. 

To prepare adults for the future requires making sure that learning 
will transfer from one setting to another. The process of adapting 
what one has learned in one context to new problems and settings in 
another is known as the transfer of learning. Many approaches to 
instruction may look the same if learners are simply tested on the 
facts they have memorized. But approaches can differ considerably in 
how they foster learning transfer (Donovan, Bransford, & Pellegrino, 

1999). Research has shown that knowledge learned only at the level of 
rote memory rarely transfers. Transfer is most likely to occur when 
the learner knows and understands both the facts and the “big pic- 
ture” — the underlying principles that can be applied to problems in 
new contexts. This understanding requires acquiring a deeper knowl- 
edge base of the basic patterns, relationships, and principles related to 
the information. For example, the fact that a learner has memorized 
the parts of a typical business memo and how to punctuate sentences 
does not mean he will know how to use this information on the job. 

To do so requires that he understand the various purposes for writing 
memos at his workplace and how to organize and tailor what is 
written for different kinds of audiences. This process is also enhanced 
by the explicit identification and development of cognitive and 
metacognitive strategies. 

How the EFF approach encourages transfer. The EFF Role Maps and 
Common Activities are important tools for promoting transfer. Using 
EFF, learners begin an instructional cycle by thinking about what they 
need to know and be able to do within and across their key life roles, 
such as within the family, at work, and in the community. In selecting 
a task to work on within one of those roles, they are guided to first 
look at the “big-picture” issues. For example, if they name writing 
memos at work as a goal, they are encouraged to examine why, to 
whom, and in what contexts they need to send memos before moving 
on to decide on a learning activity that will allow them to practice the 
basic skills associated with memo writing. While they are learning, 
they reflect on and monitor the cognitive and metacognitive skills 



Students develop flexible understanding 
of when, where, why, and how to use their 
knowledge to solve new problems if they 
learn how to extract underlying themes 
and principles from their learning 
exercises. Understanding how and when 
to put knowledge to use — known as 
conditions of applicability — is an 
important characteristic of expertise. 

— Bransford, Brown, & Cocking (1999) 

Learners are not always able to readily 
identify what they are learning when 
using a contextual approach. Teachers use 
the EFF skills wheel posted in the class- 
room to help students identify what they 
have learned. Following activities, learners 
discuss what they have learned and how 
they can apply it in the various roles that 
they play. This approach has been 
particularly useful in helping learners 
solve difficulties in their lives. The teachers 
have been able to use the skills wheel to 
help break the challenge into manageable 
pieces and prioritize. In this way, learners 
have been able to experience increased 
success in resolving personal difficulties 
while identifying skills they possess and 
need to develop. 

— Robin Stanton, Tacoma, Washington 
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For reflection. . . 

• Think of a situation where you have transferred 
knowledge or a skill learned in one context to a new 
context. What helped you to do so? 

• How do you help your learners transfer what they have 
learned? How could you use the EFF Framework to 
encourage transfer of learning? 



they are using. Once the activity is completed, learners are 
often asked to go back again to the Role Maps to investigate 
how what they have learned might transfer to other roles 
they play in life. For example, learners might be asked to 
brainstorm how what they have learned about memo writ- 
ing could apply to sending notes to a child’s teacher or to work 
they do in the community. 



I used to plan so that a specific learning activi- 
ty would take up the hour and a half that I 
spent with a student; a full circle, beginning to 
end. I felt that this was to the student’s advan- 
tage — the preview, presentation of new con- 
cepts or materials, and then closure. . .[Now] 

I like to think of activities less as a series of 
closed circles and more as an educational 
spiral. I can link one week’s lessons to the next 
simply by considering student work as moving 
toward “independent action” or attaining 
greater “voice.” With overarching views such 
as that, I can begin to think of my work with 
students as fitting a continuum of learning, 
which is the way I like to view attainment of 
literacy in general. 

— Jim Carabell, Burlington, Vermont 

As a family literacy program, we taught GED 
competencies in the context of parenting. This 
approach seems to make a lot of sense, since 
competencies break down learning into man- 
ageable chunks. Learners saw success. What 
they didn’t see was the big picture: How these 
individual successes “fit” in terms of broader 
roles. . . The switch to standards-based educa- 
tion meant focusing on the skills, rather than 
the specific context. . . Our curriculum spirals 
around the skills, revisiting them within new 
contexts in each of the three roles. We now 
measure student achievement in the ability to 
transfer skills learned in one role to another: 
the ability to apply skills across contexts. 

— Jane Meyer, Canton, Ohio 



Learning is a function not only of the activity itself but 
also of the context and culture in which it takes place. 

Lave and Wenger (1991) point out that because our lives are 
social, so are our experiences and the processes by which we 
come to understand them. Their research has shown that, far 
from being a trivial matter, the social nature of learning is a cen- 
tral aspect of education (Lave & Wenger, 1991; Wenger, 1998). 
Learning always takes place within a specific social context. The 
classroom, the teacher, the culture of the school, and the broader 
community influence how people construct their definition of 
education and what it can do for them (Street, 1999). They learn 
to develop an identity within the community at the same time as 
they master new knowledge and skills. In teacher-directed class- 
rooms where there is little interaction among students, students 
may come to see learning as something imposed by an “expert” 
rather than learning to see themselves as lifelong learners who 
construct knowledge for themselves. Within EFF-based class- 
rooms, the aim is to design a learning environment that is simul- 
taneously community centered, learner centered, and knowledge 
centered (Bransford, Brown, & Cocking, 1999; see “Designs for 
Learning Environments” on page 2). In these contexts, learners 
are encouraged to work as a collaborative team to identify and 
solve problems — just as scientists, mechanics, nurses, musicians, 
citizen group members, and parents do in everyday life. It is 
through these kinds of collaborative experiences that adults can 
come to see that learning is a process of continually transforming 
and being transformed by social experience (Gee, 1999). 

Curriculum development as an iterative process. Planning for 
contextualized learning requires that teachers make a fundamen- 
tal shift in their understanding of what it means to plan curricula 
and instruction. Instead of mapping out all the prerequisite 
knowledge and skills students need and planning lessons before 
discovering learners’ immediate needs (Nelson & Hammerman, 
1996), teachers begin with tasks learners need immediately in 
their daily lives and then “back into” the knowledge, skills, and 
strategies required to perform those tasks. This does not mean 
that basic skills are not covered, but they are addressed in an iter- 
ative rather than a sequential manner. These same processes 
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For reflection. . . 



apply to curriculum development in which skills are 
cycled and recycled across a series of tasks. This 
approach allows EFF teachers to avoid the com- 
What approach do you take to mon problem of teaching a curriculum that is 
curriculum planning? How can an “ a m q e wide and an inch deep” (Bransford, 
iterative planning approach help Brown, & Cocking, 1999) and allows learners 
you to contextualize instruction? to develop a deeper understanding of the “big- 

picture” ideas and real-life applications. 



Putting Contextualized Learning to Work in Your Program 



Results That Matter: An Approach to Program Quality Using Equipped for the Future 
(Bingman & Stein, 2001) provides a vision for program-level system reform 
(referred to as the EFF Quality Model). The EFF Quality Model identifies Pro- 
gram Practices that reflect the theoretical foundations of EFF and provides a 
guidepost by which administrators, teachers, students, and communities can 
assess their implementation of the EFF Framework. As you reflect on the examples 
below, think about how your program might answer the questions “What does it 
mean to practice EFF?” and “What does EFF implementation look like in action?” 



EXAMPLE 1: 

Teachers and students 
use the EFF Framework to 
construct contextualized 
learning opportunities that 
focus on the development 
and practice of skills the 
students need to carry out 
activities and accomplish 
purposes in their lives. 



K aren Hippert, an ABE teacher and EFF field researcher in Ohio, describes how the 
idea for a learning activity arose out of a class discussion about planning for a class 
trip. Karen and her students had been working together for some time and decided 
they would like to take a trip together. Karen knew that many of her students wanted 
to improve their math skills but often found math hard, boring, and disconnected 
from their everyday lives. She used the opportunity of the class trip to suggest to the 
group that they plan ahead to figure out how much the trip would cost. This was a for- 
eign concept for all the students in her class. None of them had ever applied their math 
skills to advance planning. Yet Karen knew that financial planning was a “big-picture” 
concept related to mathematical problem solving in many contexts. 

Karen looked at the Standard Use Math to Solve Problems and Communicate. The 
Components of Performance for the Standard helped guide her in planning the activi- 
ty: Understand, interpret, and work with pictures, numbers, and symbolic informa- 
tion; Apply knowledge of mathematical concepts and procedures to figure out how to 
answer a question, solve a problem, make a prediction, or carry out a task that has a 
mathematical dimension; Define and select data to be used in solving the problem; 
Determine the degree of precision required by the situation; Solve the problem using 
appropriate quantitative procedures and verify that the results are reasonable; and 
Communicate the results using a variety of mathematical representations, including 
graphs, charts, tables, and algebraic models. 



With this as a guide, Karen and her students developed the activity. They began by 
learning how to use a mileage chart. They applied their knowledge of multiplication 
and division to figuring out the mileage for their trip. They also spent some time 
looking at when and where estimation might be a better strategy to get at an answer 
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For reflection. . . 

• How was what these students learned 
different from what they might have 
learned by simply solving problems related 
to calcidating mileage in a math book? 

• What might Karen do next to help learners 
see how the skills they had learned might 
apply to other kinds of planning? 



quickly. Next they learned how to read and make their own graphs and charts 
to compare information related to different modes of travel and vehicles. 
Many were surprised at what they found out about the relative costs of dif- 
ferent kinds of travel. 



As they completed the activity, Karen asked them to think about how what 
they had learned might transfer to other contexts. Suddenly, Karen 
observed, it dawned on them that they could use math for all kinds of plan- 
ning. By contextualizing instruction in a real-life application, Karen’s stu- 
dents had broken through to a new understanding of the importance of math. 



EXAMPLE 2: 

Teachers use the EFF 
Framework to integrate 
“found lessons” that 
arise from in-class or 
out-of-class student 
needs into an overall 
learning plan. 



J 



For reflection. . . 

• Can you think of a situation where you were 
able to turn a real-life situation into a “found 
lesson”? 

• In what other ways can teachers use the EFF 
Framework to place “found lessons” into a 
meaningful context for adult learners? 



im Carabell (1999) describes how he helped one of his students to see how impor- 
tant and meaningful learning activities can be “found” in the events of everyday life. 
One day, after beginning a math lesson with Tammy, a 22 -year- old single Vermont 
parent working toward her GED, she mentioned that a state trooper might interrupt 
their lesson that day. She told Jim how she was in the process of trying to untangle her- 
self from the complications of buying a $500 car from her brother, who, through a 
series of events, didn’t hold the title. Jim stopped what he was doing and began helping 
Tammy to fill out the papers she had received from the trooper at the police barracks. 
Together they wrote an explanatory letter to the DMV, made a couple of information- 
al phone calls, and copied and mailed the key information to the DMV. At the end of 
their time together, Jim was able to show Tammy how much she had learned through 
this “unintentional” lesson. Tammy saw that she had achieved some of her broader 
purposes for learning. She had learned to gain access to information, give voice 
to her opinions, and act independently. What’s more, she had worked in 
some detail on three EFF Standards: Convey Ideas in Writing, Learn 
through Research, and Solve Problems and Make Decisions. As Tammy 
considered how she might use these writing, research, and problem- 
solving skills in other parts of her life, such as in her role as a parent, 
she expanded her mental model of learning and became aware that her 
time with Jim had indeed not been wasted. 



EXAMPLE 3: 

Students use the EFF 
Framework to identify skills 
learned and practiced 
through real-world learning 
activities. 



A lthough in many programs teachers and learners decide to work on a single stan- 
dard together, in other cases a contextualized learning activity can be designed to 
allow learners to address different standards while working on the same activity. For 
example, when learners at the Canton, Ohio, Even Start Program decided to set up a 
family math night for their elementary school, they divided into committees based on 
their learning goals. Octavia, who had set a math goal, volunteered to be on the budget 
committee. Rosa, who had a writing goal, served on the committee that wrote a pro- 
posal to the principal requesting permission to do the project. Eou, who wanted to 
improve her computer skills, helped create a flyer to advertise the program. After the 
project was over, the program provided learners with a form to help them reflect on 
what they could do now that they could not do before. Octavia noted that although 
she already knew how to add, subtract, and multiply decimals, she had not known how 
to use those skills to prepare a formal budget. Already she had used what she had 
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For reflection. . . 

• How did the context and culture in which 
this activity took place help learners to see 
how they could transfer what they had 
learned? 



learned to develop a personal budget at home. Rosa wrote that this had been 
the first time she had written anything that would be read by someone as 
important as a principal. She realized that she had good ideas she could 
express through writing. Learning in a real-life context had made it easier 
for these students to see how they could transfer what they had learned to 
other contexts. (Meyer, 1999) 



Glossary 

Cognitive strategies: Any behavior, thought, or action a learn- 
er engages in during learning that is intended to influence the 
acquisition, storage in memory, integration, or availability for 
future use of new knowledge and skills. (See Weinstein & 
Hume, 1998, p. 12; Pressley & Woloshyn, 1995.) 

Common Activities: The term EFF uses to refer to those activ- 
ities that adults perform in all three roles (worker, family 
member, community member). The EFF team identified the 
13 Common Activities by looking across the Broad Areas of 
Responsibility, the Key Activities, and the Role Indicators for 
each Role Map. (See Stein, 2000, p. 14; Merrifield, 2000, 
pp. 33-34.) 

Constructivism: A theory of learning and knowing that holds 
that learning is an active process of knowledge construction in 
which learners build on prior knowledge and experience to 
shape meaning and construct new knowledge. (See Lambert & 
Walker, 1995.) 

Content Standards: The term used in a variety of fields to 
describe what individuals need to know and be able to do for a 
particular purpose. In EFF, the 16 Content Standards identify 
what adults need to know and be able to do in order to meet 
their goals for learning and to be effective in their adult roles. 
Each EFF Content Standard consists of the title of the stan- 
dard and the Components of Performance for that standard. 
(See EFF Standards, Stein, 2000, pp. 19-20.) 

EFF Quality Model: A vision of what system reform at the 
program level looks like using EFF Standards. The EFF tools, 
foundational theory and research, expected program practices, 
and predicted short- and long-term outcomes are presented 
and explained in the publication Residts That Matter: An 



Approach to Program Quality Using Equipped for the Future 
(Bingman & Stein, 2001). Ordering and downloading infor- 
mation can be found at http://www.nifl.gov/lincs/collections/ 
eff/eff_publications.html. 

Iterative: A term used in research to refer to the repetition of a 
cycle of processes with an eye toward moving ever more close- 
ly toward desired results. In EFF, the term is used to describe 
how EFF has progressively refined the concepts and compo- 
nents of EFF through research, feedback from customers 
(learners, practitioners, stakeholders, and policy-makers), 
incorporation of research developments in related areas, and 
further feedback from customers. (See Merrifield, 2000, 
pp. 4, 7-8.) 

Metacognitive strategies: Metacognitive strategies consist of 
knowledge about strategies and about one’s own thinking 
processes. They are the “executive managers” of knowledge 
and include planning, monitoring, evaluating, and revising 
one’s own thinking. (See Bransford, Brown, & Cocking, 1999; 
Hartman, 2001.) 

Role Map: A publicly agreed to, explicit, consensus depiction 
of the adult roles of worker, parent/ family member, and 
citizen/community member. For each adult role, the Role Map 
provides definitions of the Broad Areas of Responsibility, Key 
Activities, and Role Indicators, which describe, not prescribe, 
effective performance in the role. (See Stein, 2000, pp. 8-13.) 

Transfer of learning: The ability to extend or adapt what has 
been learned in one context to new problems and settings. 
Research has shown that when a subject is learned in multiple 
contexts, with opportunities to abstract general principles, 
transfer to new situations is increased. (See Bransford, Brown, 
& Cocking, 1999; Stein, 2000, p. 20.) 
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HANDOUT 2.10 

Constructivist Learning 

Building Expertise 

Constructivism, in educational theory and practice, means that learning is a process of constructing new 
meanings in which prior knowledge and experiences are the basic building blocks for building expertise. 
Constructivism is a theory of learning and of knowing. For additional background information, see EFF 
Research to Practice Note 2 in the Appendix. From conception, the human brain accumulates and stores 
knowledge, information, and skills which become the basic building blocks for understanding and mak- 
ing sense of the world. As knowledge and experience are added, a person gains new understanding and 
perspective. A teacher who approaches teaching and learning from a constructivist standpoint starts by 
finding out what learners already know and have experienced that relates to a topic or idea and helps the 
students build on that knowledge and experience to gain new concepts, information, and skills. The 
teaching and learning process is interactive and dynamic. Review the chart below which illustrates a con- 
structivist learning process. 



Constructivist Learning Process 



Student’s prior knowledge, 




New Information 






skills, background, and 




and Experiences 




New 


experience 




• Reflections 




Meanings 


• Language 


+ 


• Information 




• Social Skills 


• New Skills 




and 


• Phonemic Awareness 




• Knowledge 




• Sense of Community 

• Historical Perspective 




• Analysis 

• Practices 




Knowledge 



Constructivist teaching and learning is a 

process of discovery. It is one in which teachers 
and students use the EFF framework to identify, 
reflect on, and revise their own mental models of 
adult role performance. For example, the EFF 
Standards show the various components of a 



particular skill. The students and teacher can review 
those components of performance and decide what’s 
already known about the skill and what needs to be 
learned or improved. This review and reflection 
process is the first step in constructivist teaching 
and learning. 



AN EFF HANDBOOK FOR PROGRAM IMPROVEMENT 

121 





HANDOUT 2.11 

EFF Teaching and Learning Cycle 




STEP 1. Determine 
individual learner’s 
goals and purposes 
and identify the 
Standards that will help 
him/her achieve them. 
Identify student’s prior 
knowledge about these 
goals and Standards. 




The 

Teaching/Learning 

Cycle 




STEP 6. Observe and 
document evidence of 
performance of the 
Standard. 



STEP 2. In a group identify 
a shared interest, purpose, 
or goal and determine the 
group’s prior knowledge of 
this topic. Identify the 
Standard that will help the 
group address this shared 
goal. Make clear the 
connection between the 
class focus and individual 
needs. 




STEP 3. Design a learning 
activity to address the real-life 
concerns of the learners. 



STEP 4. Develop a plan 
to capture evidence and 
report learning. 
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HANDOUT 2.12 

EFF Teaching and Learning Cycle Grid 



EFF Teaching/Learning 
Cycle Steps 


Implications for 
Administrators 


Changes or 

Program Improvements 


1 . Determine individual learner’s 
goals and purposes and 
identify the Standards that 
will help him/her achieve 
them. Identify student’s prior 
knowledge about these 
goals and Standards. 


For example: 

Students complete an 
interest survey. 


For example: 

Discuss student goals 
at intake and share with 
teachers who will use to 
plan with students. 


2. In a group identify a shared 
interest, purpose or goal and 
determine the group’s prior 
knowledge of this topic. 
Identify the Standard that will 
help the group address this 
shared goal. Make clear the 
connection between the class 
focus and individuals’ needs. 






3. Design a learning activity to 
address the real-life concerns 
of the learner(s). 






4. Develop a plan to capture 
evidence and report learning. 






5. Carry out the learning activity. 






6. Observe and document 
evidence of performance 
of the Standard. 






7. Evaluate and reflect on 
how what was learned is 
transferable to real-life 
situations? 






8. Determine next steps to help 
learners meet their goals. 
(Return to Step 1 and/or 2) 
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HANDOUT 2.13 

Active, Purposeful and Contextual: Assessment 
in the EFF Classroom 

— by Joan Benz 



I am an instructor of adult basic skills and GED at the 
Bethel Family Learning Center in Eugene, Oregon. 
Over the last two years I have been involved with 
teaching and assessing using the Equipped for the 
Future (EFF) Framework. EFF, an initiative of the 
National Institute for Literacy, was developed to 
answer the complex question: What do adults need to 
know and be able to do in order to carry out their 
roles and responsibilities as workers, parents and fam- 
ily members, and citizens and community members? 
(NIFL, Equipped for the Future Content Standards, 
2000). EFF Standards have been identified through a 
careful research process that began with adult learners 
and has included administrators, practitioners, tutors, 
and policy makers as well as experts from adult educa- 
tion, literacy, workforce development, and other stake- 
holder systems. 

The 16 EFF Standards represent the core skills needed 
for effective adult performance in the three major 
adult roles in today's rapidly changing world and are a 
new definition of literacy for the 21st century. The EFF 
Standards Framework includes: 

• Four purposes of learning defined by adult students: 
Access to Information, Voice, Independent Action, and 
Building a Bridge to the Future. 

• Role maps that define what effective adults need to 
know and do to carry out their responsibilities. The 
three role maps are worker, parent/family, and citi- 
zen/community member. 

• Common activities that cross all three roles. 

• 16 standards that support effective performance in 
the three roles to achieve the four purposes. 

EFF is very exciting to use in class because it is based 
on input from adult learners and therefore is very 
meaningful to my students. Learning in an EFF class- 
room is active, purposeful, and contextual. Students 
are very much in control of their own learning. 



Classroom activities are developed around perfor- 
mance tasks. Performance tasks are real-life activities 
that allow students to demonstrate performance of 
one or more of the EFF Standards. An example of a 
performance task would be a group of activities that 
students would do to be able to convey ideas in writ- 
ing (an EFF Standard) in a letter to their child’s 
teacher. Learning activities would address components 
of performance or skills needed to be able to use the 
standard for a meaningful purpose. Here are the com- 
ponents of the Standard, Convey Ideas in Writing, 
(NIFL 2000): 

• Determine the purpose for communicating. 

• Organize and present information to serve the pur- 
pose, context, and audience. 

• Pay attention to conventions of English language 
usage, including grammar, spelling, and sentence 
structure, to minimize barriers to reader’s compre- 
hension. 

• Seek feedback and revise to enhance the effectiveness 
of the communication. 

A well-structured performance task will address all 
these components. 

There is still a point to consider. How do I know that 
my students are learning? A paper and pencil test will 
not capture what these students know and are able to 
do. The EFF Framework addresses assessment as 
movement along a continuum of learning. As people 
learn, they increase their knowledge, fluency, indepen- 
dence, and range in using a skill. EFF refers to these as 
four dimensions of performance. Each dimension 
helps describe not only what people know, but also 
how well they can use what they know (NIFL, 2000). 

This is where, for me, instruction and assessment 
combine. Using the four dimensions of performance 
allows me to think about what skills are needed to per- 
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form the task and to look at where student skill levels 
are related to these dimensions before and after the 
task. 

Knowledge Base 

• What vocabulary do learners have related to the 
skill? 

• What content knowledge to the learners have related 
to the skill? 

• What strategies do learners have for organizing and 
applying content knowledge? 

Fluency 

• How much effort is required? 

Independence 

• How much help is needed from others? 

Range 

• In how many different contexts can learners per- 
form? 

• How many different tasks can the learner do using 
the skill? 

One way I learned to better understand these dimen- 
sions of performance was to think about something I 
was learning: in my case it was learning to swing 
dance. 

When I first started out I really didn’t have much of a 
knowledge base — just what was in the catalog. As I 
learned, I picked up vocabulary (basic step, loop pass, 
etc.) and got better at organizing what I was learning 
by being able to put these steps together. Fluency was a 
big problem when starting out. I had to count and 
concentrate on each step. As I got better, some of the 
steps became automatic. At first I had a hard time 



learning from watching and needed the instructor to 
demonstrate the steps by being my partner. My inde- 
pendence grew as I gained confidence and didn’t need 
as much “hands on” support. My range of perfor- 
mance is still limited. I haven’t danced anywhere other 
than the classroom. My goal is to be able to dance (and 
enjoy it) at my son’s wedding next summer. I have a 
ways to go, but I can see that I am learning and 
improving. 

This is a lot of information for a short article, and I 
have to say that I certainly don’t have all the compo- 
nents working together smoothly in my classroom yet. 
However, using EFF as a framework for assessment 
and instruction has allowed me to become a more 
intentional and informed instructor and learner. 

The late Joan Benz was an instructor of adult basic skills 
and GED at the Bethel Family Learning Center in 
Eugene, Oregon. EFF is an initiative of the National 
Institute for Literacy. For more information about EFF, 
visit www.nifl.gov/lincs/collections/eff/eff.html 
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Adult Basic Education Support and funded by the fed- 
eral Adult Education Act (S.353), administered by the 
Massachusetts Department of Education, Adult and 
Community Learning Services (ACLS) Unit. 
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HANDOUT 2.14 

Assessment, Accountability and Program Improvement: 
Background and Case in Point 4 



Background: 

In a quality program, assessment and instruction are aligned. Administrators need to know 
how student assessment is currently done and what needs to happen to move towards an approach that is 
grounded in cognitive science research on the development of expertise. In order to support the alignment 
of instruction and assessment, the quality improvement team reviews current assessment practices and 
compares them with assessment practices based on the EFF Framework. The team also reviews what and 
how the program collects data and documents student progress in order to identify changes that would 
improve organizational performance and outcomes. 

The question for the team is: 

How will this agency align assessment with student and program goals and document 
student progress and change? 



Case in Point: 

Students enrolling in the Pierce CommunityBuild Adult Education Center first take a test to determine their read- 
ing and math levels. Based on the results of the test, students are placed either in the low-level class or the high- 
level class. At the end of the quarter, students are re-tested to see if their skills in reading and math have increased. 
Test data goes to the local community college because they provide some funding to the CommunityBuild Center. 
The test results help to inform how the community college allocates its resources to various community based 
organizations like CommunityBuild. 

Ms. Franklin teaches the low-level classes and has observed how test results affect the attitudes of many students. 
Some of the students feel shamed by the test results and become discouraged. After seeing their low scores, some stu- 
dents feel that they have too much to learn and that they will never be able to earn a GED. 

Ms. Franklin has been experimenting with the EFF Framework, selecting a Standard that will help the group of 
students address a real life goal. Recently, the class reviewed the Standard, Take Responsibility for Learning. This 
Standard was relevant to the students’ attitudes about learning since they all wanted to take responsibility for their 
learning. Ms Franklin explained that there are several dimensions on which students need to perform in order to 
master a standard: the knowledge base, the fluency, the independence, and the range. The class had a lively discus- 
sion defining what these dimensions of performance mean to them. 

Ms. Franklin had each student create a matrix that listed all of the Standard’s components in the vertical column 
and the four dimensions of performance in the horizontal row. They each selected a particular skill that they want- 
ed to master and did an initial assessment of their knowledge, fluency, independence and range. After three weeks 



AN EFF HANDBOOK FOR PROGRAM IMPROVEMENT 

127 



they created the same matrix to reflect on their progress. The students embraced this way of assessing their 
progress and felt that it more accurately measured and reflected their learning and abilities. Here is an example of 
the matrix. 



Take Responsibility 
for Learning 


Knowledge 


Fluency 


Independence 


Range 


1. Establish learning 
goals... 










2. Identify own 
strengths... 










3. Become familiar with 
a range of learning 
strategies... 










4. Identify and use 
strategies appropriate 
to goals... 










5. Monitor progress 
toward goals... 










6. Test out new learning 
in real-life applications. 
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HANDOUT 2.15 

Questions on Assessment 

What kinds of learning are measured?(skills, competencies, etc.) 



How is learning measured? 



What evidence is gathered to evaluate performance? 



How is this evidence used? 



Is the learning scored? If so, how? 



What is the student’s role in the assessment process? 



How does assessment connect to instruction? 



How does assessment connect to curriculum and content standards? 



How does assessment connect to education theory and research? 



How are assessment results documented? 
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HANDOUT 2.16 

Current Approach to Assessment 



Questions 


Current Approach 


What kinds of learning 
are measured? (i.e. skills, 
competencies, etc.) 




How is learning measured? 




What evidence is gathered 
to evaluate performance? 




How is this evidence used? 




Is the learning scored? 
If so, how? 




What is the student’s 
role in the assessment 
process? 




How does assessment 
connect to instruction? 




How does assessment 
connect to curriculum and 
content standards? 
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HANDOUT 2.1 7 

Traditional Versus EFF Approaches to Assessment 





Current Approach 


EFF Approach j 


Construct (what is 
measured) 


Discrete knowledge, skills, or 
competencies 


Integrated knowledge, skills, 
strategies, and abilities 


Task format 
(how it is measured) 


Questions or problems posed in 
isolation or with a small amount of 
supporting context, tasks (test items) 
usually of short duration, and typically 
require recall and analysis 


Task situated in activity that closely 
mirrors real-world activity, may take 
place over an extended period of time, 
and can involve creativity, strategic 
thinking, and problem-solving 


Response format 
(evidence used to 
evaluate performance) 


Selected response or short, written 
response, usually collected at one time 


Constructed response, oral, written, 
graphic, and/or behavioral response or 
performance that may be collected 
over a period of time (as in a portfolio) 


Scoring format 


Scored as right or wrong or on a 
one-dimensional scale (assigned a 
letter or number grade) 


Scored with a rubric that identifies 
dimensions of performance and 
qualitative differences in levels of 
performance 


Role of student 


Student are not told what the questions 
will be before the test and typically do 
not assess their own performance 


Students know in advance the nature 
of the task and how their performance 
on the task will be evaluated and can 
self-assess and monitor their own 
performance 


Connection to 
instruction 


Separate from instruction but tests 
discrete skills or knowledge that are 
the objects of instruction 


Embedded in instructional and 
learning activities or on-demand tasks 
that have the same structure as 
instructional and learning activities 


Connection to 
curriculum and 
content standards 


Assessed knowledge, skills, and 
abilities may be poorly or incompletely 
aligned with curriculum and content 
standards 


Assessed knowledge, skills, and abili- 
ties are aligned with standards and 
standards-based curricular content 


Connection to theory 
and research 


Content and structure of assessments 
may be related to skill hierarchies but 
are typically not derived from a theory 
of learning, development, or expert 
performance 


Content and structure of the 
assessments are derived from 
analyses of adult learner 
performance, cognitive science 
models of developing expertise, 
and adult learning theory 
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HANDOUT 2.18 

Performance Accountability: For What? To Whom? And How? 

— by Juliet Merrifield 



In everyday life, accountability means responsibility; it 
means being answerable to someone else for one’s 
actions. We cannot, however, use the term without 
specifying accountability to whom and for what. In 
adult basic education (ABE), how we answer the ques- 
tion “to whom” depends a lot on our position in the 
system. Teachers may answer that they feel answer 
accountable to their students. Program directors may 
that they are accountable to their funders and staff as 
well as to students. State adult education offices may 
feel accountable to the governor, the legislature, to 
other state agencies, to workforce development boards, 
as well as to taxpayers. In addition, no clear consensus 
exists about “for what” adult education is accountable. 
Where does the balance lie between providing services 
and delivering results? Is the main purpose increased 
literacy proficiency, or are more diffuse social out- 
comes the emphasis? Until recently, the focus has been 
on providing services, with little emphasis on the 
results or the impact of those services. In the last few 
years, a number of policy initiatives at state and federal 
levels have begun to shift the emphasis to delivering 
results, with services seen as the means to an end. But 
what the “end” should be is by no means clear. 

I would like to suggest that developing performance 
accountability is not just technically challenging but 
also challenges our values. The key issues do not have 
purely technical solutions. They require agreement on 
what is important to us, on what we want out of adult 
education. If they are to be resolved, they require 
involvement by the ABE field as a whole. 

Adult basic education is facing serious demands from 
policy-makers and funders to be accountable for its 
performance. The 1998 Workforce Investment Act 
(WIA) requires that each state report on performance 



measures. The emphasis on results shifts attention 
from simple delivery of services to the outcomes of 
learning: learning gains measured on standardized 
tests or social and economic outcomes such as getting 
a job, getting off welfare, and children’s school success. 
The key issues in the development of performance 
accountability in adult education are: 

• What does good performance mean? 

• Do programs have the capacity to be accountable? 

• Are the tools commonly used for measuring and 
documenting performance adequate and useful? 

• Are accountability relationships in place to link ABE 
into a coherent system? 

Good Performance 

Accountability systems work best if stakeholders — 
those who have an interest in the outcomes of the sys- 
tem — agree on what success looks like. For adult basic 
educators, the heart of the matter is our concept of lit- 
eracy. That concept has shifted over time from reading 
and writing text to functioning in society, from a simple 
dichotomy of illiterate/literate to multiliteracies. Brian 
Street characterizes two broad conceptual notions of 
literacy. The autonomous model conceives of literacy as 
a discrete and fixed set of skills, transferable from one 
context to another. The ideological model conceives of 
literacy as practices that are sensitive to social context 
and inherently associated with issues of power and 
access (Street, 1984). 

Much recent research on multi-literacies suggests that 
there are multiple purposes for literacy and multiple 
goals and expectations for literacy education (Heath, 
1983; Barton, 1994; Street, 1984,1995; Lankshear, 
1997; New London Group, 1996). In such an under- 
standing, notions of success must also be multiple. A 



Article reprinted from Focus on Basics, Voi. 3, Issue B, June 1999 and based on Contested Ground: Performance 
Accountability in Adult Basic Education, NCSALL Report #1 by Juliet Merrifield. The full report can be downloaded from 
http://gseweb.harvard.edu/~ncsall/research/reports.htm 
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single definition of success — gaining the GED, for 
example, or getting a job — excludes learners who have 
different purposes. 

Definitions of success should be negotiated among all 
the stakeholders, learners, and practitioners as well as 
policymakers and funders. Although the legislative 
goals of the Workforce Investment Act reflect a major- 
ity among lawmakers, other stakeholders — including 
policy makers, program managers, teachers and stu- 
dents — may focus on other purposes for adult educa- 
tion and look to other measures of good performance. 

Next Steps: Agree on Performance 

Practitioners can play a role in defining performance 
within their own states. The WIA requires that each 
state develop a plan of the performance measures it 
will use to track results, including but not limited to 
those required by the Act. Whether explicitly or 
implicitly, these measures will define what counts for 
the field. The challenge is to come to an agreement on 
performance that includes the full diversity of learner 
and societal purposes. Lessons from the literature and 
experience in education and other fields suggest states 
should: 

• Invest time and energy in agreeing on what perfor- 
mance means; 

• Involve stakeholders and seek consensus; 

• Reflect newer understandings of literacy and con- 
nect performance with real life; and 

• Acknowledge a variety of outcomes as acceptable 
performance, as a way of including the full diversity 
of learners and programs. 

Capacity to be Accountable 

Adult education is trying to develop a national 
accountability system without having developed the 
capacity of the service delivery system to document 
and report results (Moore & Stavrianos, 1995). Plenty 
of evidence documents the lack of valid, reliable, and 
useful data about performance (Young et ah, 1995; 
GAO, 1995; Condelli, 1994). These studies suggest 
some of the most basic data are absent, incomplete, or 
of low quality. 



When asked to report numbers, programs will indeed 
report numbers. But as the GAO report on adult edu- 
cation says, “the data the Department receives are of 
questionable value” (GAO, 1995, p. 33). This is not 
surprising, since staff in programs usually do not use 
the data, rarely see reports based on them, and see no 
one else placing any real value on them. 

Performance accountability requires investment in the 
ability of local programs to collect, interpret, and use 
data to monitor how well they are doing. A number of 
states such as Pennsylvania, Connecticut, and Arkansas 
have already begun to develop their capacity for 
accountability. (For an overview of Pennsylvania’s pro- 
gram, Keenan’s article.) They consistently learned from 
their experiences that the key is to get buy-in from pro- 
grams and practitioners from the beginning (Merri- 
field, 1998). They are also acutely aware of the 
problems of deciding what is counted, as well as how it 
is counted. 

What is counted becomes what counts. Many exam- 
ples of the hazards of counting the wrong things exist. 
A healthcare delivery system emphasizes cutting the 
numbers of people on a waiting list for surgery, thus 
ensuring that people with minor needs get served 
quickest because more operations for varicose veins 
than for heart bypasses can be performed in one day. 
The original performance standards of the Job Train- 
ing Partnership Act (JTPA), an education and training 
program, emphasized the numbers of people placed in 
jobs within a specific time frame. This ensured that 
programs recruit clients who were most qualified and 
therefore easiest to move into jobs quickly and cheaply 
(GAO, 1989). 

Next Steps: Build Capacity 

Two kinds of capacity — to perform and to be account- 
able — are linked (Merrifield, 1998). By instituting a 
learning organization approach with feedback loops, 
performance data can help programs improve perfor- 
mance and increase accountability. Building the 
capacity to perform involves: 

• Increasing resources and focusing them on quality 
rather than quantity; 
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• Providing staff development and training and tech- 
nical support; 

• Using performance data for continuous improve- 
ment. Building the capacity to be accountable 
involves ensuring that: 

• Accountability demands are commensurate with 
resources and capacity; 

• Users of measurement tools are engaged in their 
development; 

• Staff training and support are provided; 

• Information is timely; 

• Improved performance is rewarded. 

A variety of efforts are already underway to build 
capacity to perform and to be accountable. Teacher 
inquiry projects have involved individual teachers in 
examining their practice and identifying ways to 
change and develop (Smith & Lytle, 1993). Some pro- 
grams, such as those described elsewhere in this issue, 
have been working on their capacity to use data for 
continuous improvement. Some states have begun 
efforts to build local program capacity for both per- 
formance and accountability. The National Account- 
ing and Reporting System (Condelli, 1998), will be 
providing training and support on how to use newly 
revised WIA- related performance measures. 

Accountability Tools 

For accountability purposes, it is crucial that we col- 
lect data that are relevant, adequate, and important. To 
do so, we need tools — indicators and measures — that 
we believe in and use well. Indicators and measures are 
approximations of reality, not reality itself. They can 
be good, bad, and indifferent. An indicator that mea- 
sures something unrelated to literacy learning — the 
number of brown-eyed learners, for example — is irrel- 
evant. An indicator that measures something rele- 
vant — the prior learning that students bring, for 
example — but in an inadequate way, is dangerous. An 
inability to measure something important — affective 
changes in learners, for example — can be disastrous. 

Some of our current accountability tools are inade- 
quate: what we use to measure literacy gains is one 
example. Standardized tests are widely used. While such 
tests have their uses for placement purposes, their valid- 



ity as measures of performance is questionable 
(Venezky, 1992). “The research literature raises ques- 
tions about the validity of standardized tests... and local 
program staff have questioned the appropriateness of 
using these assessment to measure program results” 
(GAO, 1995, p. 24). As yet, however, few alternatives to 
standardized tests exist. Some programs are using vari- 
ous tools, such as portfolios, that allow learners to 
demonstrate their learning authentically (Literacy 
South, 1997), but so far these cannot compare learning 
between learners and across programs. Without exter- 
nal criteria or standards, authentic assessment will not 
meet the needs of accountability systems. 

How we collect data for accountability is also impor- 
tant. Different approaches to data collection and 
analysis meet different purposes. A complete perfor- 
mance accountability system would include several 
approaches: monitoring, evaluation, and research 
would all have a place. 

Monitoring can answer ongoing questions about day- 
today program operations. What kinds of students are 
being recruited? How long are they staying? What do 
they say they want from their learning experiences? 
How satisfied are they with the program? Monitoring 
is part of everyday management, providing a routine 
way for program staff to see how well the program is 
working. 

Evaluation can answer particular questions about pro- 
gram operations at particular points of time. How are 
learners being served? Are they making progress on 
their learning goals? Is the program meeting quality 
standards? Evaluation may include a look at program- 
monitoring data. It may also involve gathering new 
data to answer specific questions. Surveys or focus 
groups are useful evaluation techniques. 

Research can answer questions about associations, 
correlations, and meaning, and often takes a broader 
focus than one program. Research questions might 
examine: What are the benefits to individuals and 
society of participation in adult education? Which 
program designs are associated with different results? 
What kinds of resources are needed to support specific 
program designs? Research may be conducted by out- 
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side researchers or by practitioners themselves 
(Quigley & Kuhne, 1997). 

Each of these accountability technologies illuminates 
different aspects of reality. They have different 
strengths and need to be used appropriately. Carrying 
them out involves scarce resources, so they should be 
applied carefully and economically to ensure that the 
data collected are both useful and used. 

Next Steps: 

Develop New Measurement Tools 

New approaches and tools for measurement are need- 
ed that are linked to performance. Performance assess- 
ment tools enable us to assess literacy practices. For 
accountability purposes, this more authentic assess- 
ment of literacy practices demands that we develop 
external standards or criteria against which individual 
student learning can be measured, and through which 
program performance can be assessed. Initiatives in 
performance assessment in countries such as Britain 
and Australia may provide useful models for measur- 
ing and assessing learning. We should use the full 
potential of research, evaluation, and monitoring tech- 
nologies to meet the needs of different stakeholders. 

Mutual Accountability 

Underlying all the other issues in performance 
accountability for ABE is the question of accountabili- 
ty relationships. Traditional approaches to account- 
ability echo Taylorist manufacturing systems, in which 
quality control checks at the end of the production 
line ensure that widgets meet product specifications 
and accountability runs only one way. Assessing out- 
comes at the end of the production process has its 
place in quality control systems, but increasingly busi- 
nesses are turning to more participatory approaches to 
managing work processes and using production data 
for continuous improvement (Stagg, 1992). 

High performance workplaces build in processes at 
each stage of production to monitor and improve per- 
formance. They involve workers in this monitoring. 
The business world is now utilizing concepts such as 
the learning organization: one that facilitates the 



learning of its members to transform itself continu- 
ously (Pedler et al, 1991). This approach is seen as a 
way of responding to changing environments and 
multiple demands. This kind of learning and transfor- 
mation has to be shared and internalized: it cannot be 
imposed from the top (Stein, 1993). Accountability is 
shared or mutual. 

In ABE, mutual accountability would engage members 
of the organization in creating a common vision, 
determining goals and customer expectations, and 
designing effective means of monitoring processes and 
results. Every member would be both accountable to 
others and held accountable by them. Learners would 
hold teachers, for example, accountable for providing 
learning opportunities that meet their needs. Teachers, 
in turn, would hold program directors and funders 
accountable for providing the resources they need to 
meet learner needs. These might include materials, 
space, training, pay for lesson planning and assess- 
ment. 

Spelling out relationships of mutual accountability 
reveals some that are overlooked in conventional 
accountability systems. Congress, for example, holds 
adult education programs accountable for providing 
effective and efficient services. But Congress should 
also be held accountable by programs, by learners, and 
by voters for identifying a social need, passing appro- 
priate guiding legislation, and providing the resources 
needed to create a strong adult education system. 

Learners should hold their teachers accountable. But 
programs should also hold learners accountable for 
taking learning seriously and for making an effort to 
participate fully. 

Businesses who expect adult education to provide 
them with workers equipped with basic skills might be 
expected in turn to provide jobs for those workers, or 
to continue a workplace basic skills program when the 
grant runs out. Mutual accountability would require 
all the partners to honor their contracts. 

An accountability system based in the concept of 
mutuality has several characteristics: 

• It is negotiated between the stakeholders in a process 
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that engages all the players in clarifying expecta- 
tions, designing indicators of success, negotiating 
information flows, and building capacity. 

• Each responsibility is matched with an equal, 
enabling right: the right to a program that meets 
one’s learning needs with the responsibility to take 
learning seriously, for example. 

• Every player knows clearly and agrees to what is 
expected of them. 

• Every player has the capacity to be held and to hold 
others accountable. 

• Efficient and effective information flows enable all 
players to hold others accountable. 

Inequalities of power and uneven access to informa- 
tion prevent the development of mutual accountabili- 
ty. Learners, for example, cannot become real 
stakeholders in mutual accountability until they have 
other ways to effect change beyond dropping out. 
They will only become part of the structure of 
accountability when they have real power to make 
choices. Some community- based programs encourage 
learner participation in management, with learner 
representatives sitting on boards, and being involved 
in management decisions about the program. Many 
state-level adult learner organizations are working to 
address the inequalities in power and in access to 
information, and to strengthen the voice of adult 
learners in the system. 

How information flows is also a central issue in mutu- 
al accountability. Without adequate access to informa- 
tion, stakeholders cannot hold others accountable. In 
traditional information flow designs, information is 
collected at the base and increasingly summarized for 
the purposes of different levels on the way up: from 
program to community, state, and national levels. In 
this simplistic model, information flows only one way: 
up the system to the state and national levels. Few peo- 
ple have either access to or the ability to use the data. 

This model will not fit the needs of an accountability 
system that takes into account different performances 
and purposes and has mutuality as an underlying 
assumption. A more complex information model 
should allow information to be generated at all levels 



and to flow around the system, up, down and across it, 
among and between different players who use it for 
specific purposes at specific times. 

Next Steps: 

Develop Mutual Accountability 

Reforming accountability requires moving from one- 
way, top- down lines of accountability to a mutual web 
of accountability relationships. To make this switch, 
we must: 

• Bring the full range of stakeholder groups into the 
process — including teachers and learners; 

• Provide support for stakeholders who have least 
access to information and power; 

• Increase information flows among and between all 
stakeholders and make the information transparent 
(accessible to all); 

• Develop learning organizations at the program and 
state levels that would emphasize learning and con- 
tinuous improvement, shared responsibility, and 
engagement in monitoring results. 

What Next? 

To implement performance accountability well 
requires agreement on good performance, capacity 
both to perform and be accountable, new tools to 
measure performance, and a strong system of mutual 
accountability relationships. In the business world, 
high performance is associated with extensive changes 
in organizational practices, including a broadly under- 
stood vision and mission, flatter hierarchies with deci- 
sion-making pushed as close to the shop floor as 
feasible, and participation at all levels of the organiza- 
tion in monitoring and improving performance. If 
ABE is to meet society’s need for high performance, it 
too needs to change. But these changes cannot be 
implemented from the top alone. They will require 
federal and state government departments to consult 
with the field and with stakeholders. They need will- 
ingness to learn lessons from the past and from other 
countries. They demand a commitment of resources 
to building the capacity of the field. Above all, they call 
for the contributions of all players, practitioners and 
learners as well as policymakers and researchers. 
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HANDOUT 2.19 

Accountability, Change and Quality: 
Background and Case in Point 5 



(Adapted from "How Are We Doing?" NCSALL Teaching and Training Materials) 

Background: 

A systematic approach to continuous program improvement means having clear organizational goals and pur- 
poses that align with student goals and national policies. It also means having a process for documenting and 
monitoring not only student progress and outcomes but every aspect of the organization’s performance. Perfor- 
mance accountability, an important aspect of program improvement, is a way of conveying to stakeholders 
“what” and “how” an organization is doing. 

In this session, the quality improvement team builds expertise through a process that examines how organiza- 
tional changes have occurred in the past, what changes would lead to a higher quality program in the future, and 
identifies changes that would improve organizational performance and outcomes. 

A quality improvement program, such as the Baldrige National Quality Program, offers a framework for approach- 
ing quality improvement in an organization. Improvement processes require organizations to have a system for 
collecting and analyzing data on student progress and using it to review the effectiveness of program components 
and revise program goals. 

The team might ask: How will this agency take a systematic approach to program improvement? 



Case in Point: 

Mr. Johns, the education coordinator at the CommunityBuild Adult Education Center, returned from the 
state’s annual adult basic education conference excited about the possibility of receiving additional funds 
for the Center’s programs. At the conference, he learned about the state’s new accountability require- 
ments. To meet the accountability requirements and access the new funds, the Center needs to collect 
more information on the progress of students. 

Mr. Johns spoke to Ms. Franklin about the new requirements, and she flatly stated, “I just don’t have time 
to do more paperwork if I’m to really do my job as a teacher.” Mr. Johns understood Ms. Franklin’s 
dilemma. Mr. Johns then spoke with the Center’s Director about the possibility of getting new funds for 
the adult education program. The director wanted to know how the new accountability requirements 
help to meet the needs of the students. 

Mr. Johns was a bit overwhelmed by what he was hearing and decided to raise the issue in the next staff 
meeting. At that meeting, all of the staff agreed that the organization needed to be strategic in making 
program changes and that the entire organization needed to determine how to strengthen program 
processes and outcomes. One staff person summed it up this way, “One person can’t do it alone; it takes 
the whole staff and anyway, it’s all about helping students to meet their goals.” 
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HANDOUT 2.20 

Questions About Change 



What was the change to be made? What was (were) the goal(s) of the change? 



What were the concerns or issues to which the change responded? 



Who initiated the change? 



What was your role in the change process? Who else was involved? 



What was the plan for carrying out the change? What was the time frame? 



What skills were essential to the change process? 



What were the outcomes from the change? 



What did you observe about this particular change process? 



What could have happened to produce different outcomes? 



What important lessons were learned from the change experience? 
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HANDOUT 2.21 

Education Criteria for Performance Excellence 
and EFF Program Practice 



EFF 

Baldrige 9 ualit y 

Criteria P r °9 ram 

Practice 


Purposeful and 
f transparent 


Contextualized 

Approach 


Constructivist 

Approach 


Assessment 
Based on 
Cognitive 
Science 


Systematic 
Approach to 
Program 
Improvement 


Leadership 


r 

Program leaders 
must understand 
the organizational 
purposes and 
goals. 


How can 
students build 
leadership 
skills through 
contextualized 
teaching and 
learning? 








Strategic 

Planning 




Our program 
must be 
thoughtful and 
strategic when 
planning to 
implement 
instruction that 
is contextualized. 








Customers 


How do our 

students 

understand 

the program’s 

purposes/goals 

and how do 

understand 

student’s 

purposes 

and goals? 










Information and 
Analysis 


What information 
do we need to 
collect and 
understand? 










Human 

Resources 












Process 

Management 












Results 
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HANDOUT 3.1 

Steps in Phases 3 and 4 



From EFF Standards Solve Problems and Make Decisions and Plan 

PHASE 3 

• Evaluate strengths and weaknesses of alternatives. 

— revisit the initial reasons for engaging in quality improvement 

— revisit the organizational and program goals 

— prioritize the possible changes 

— consider the political, partnership, personnel, and financial realities of the organization 

• Select alternative that is most appropriate to goal, context, and available resources. 

— decide which program practice(s) to address and which changes to make 

— think about how the plan will be administered and the necessary funds 

• Develop an organized approach of activities and objectives. 

— determine what actions will be necessary to achieve these changes 

— think of all the people and organizations that need to be involved 

— outline the plan’s specific activities, responsibilities, a timeline for implementing 
the changes, and criteria for evaluation 



PHASE 4 

• Actively carry out the plan. 

— identify the responsibilities of team members 

— give someone the responsibility for making sure these supporting activities take place 

— be certain everyone knows their role 

— use your timeline to stay on track 

• Monitor the plan’s progress while considering any need to adjust the plan. 

— hold regular team meetings 

— document of the implementation of the plan 

— provide regular opportunities for review 

— record observations of unexpected occurrences 

• Establish criteria for evaluating effectiveness of solution or decision. 

— examine the documentation you have collected 

— identify the indicators of progress 

• Evaluate its effectiveness in achieving the goals. 

— collect data 

— analyze your data 
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